My agents said 'done' when it wasn't. I built a fleet that counts those lies. It didn't place.

Carson Rodrigues

Carson Rodrigues / November 21, 2026

7 min read • ––– views

hackathonsai-agentsgoogle-cloud
The Attest Fleet dashboard showing a silent failure rate of 4.8 percent, with reported success at 52.5 percent and verified success at 50.0 percent

An AI agent tells a customer "your refund is issued." The payment system never sent it, nobody checks, and the dashboard stays green.

HackathonAll Things Agentic, Fortified Enterprise Fleet track
OrganiserGoogle, on Devpost
When3 to 31 August 2026 (I submitted on 30 August)
Field12,265 registered
Result and rankDid not place. 16 winners were named; no ranks were published beyond the winners
Built inAbout 5 days (first work 25 August, submitted 30 August)
TeamSolo
Money spentNot tracked. My early test runs used Google's free tier; I did not record cloud or model costs. Everything on Google Cloud was deleted on 25 October, so nothing bills now
Codegithub.com/rodriguescarson/attest-fleet

That was the problem I built for in Google's All Things Agentic hackathon, which ran on Devpost from 3 to 31 August 2026. It had 12,265 registered people and three tracks. I entered the Fortified Enterprise Fleet track, the one about running many agents inside a company without losing control of them. I submitted on 30 August.

Judges scored three things: how useful the agent was and whether it acted on its own (40%), how well it was built on Google's tools (30%), and the demo and how ready it was for real use (30%).

What I built

Attest Fleet is a small customer-support team made of five AI agents. One reads a ticket and splits it into tasks. One handles billing. One handles accounts. One reads photos that customers attach. A fifth, running on a different model family, acts as an auditor.

The idea is simple. When a worker says "done," I treat that as a claim, not a fact. A separate checker then looks at the real database and asks whether the world actually changed.

If the agent said "done" and the record disagrees, that is a silent failure. I count it and publish the number.

A few more rules sit around that. A refund over a set limit stops and waits for a human. There is a kill switch that stops the whole fleet. Each run keeps an evidence trail, so you can see exactly where a claim and the record split apart.

Diagram of Attest Fleet: a ticket passes a safety screen, agents act under a policy gate on a database, a verifier compares each claim with the real record, and failures become lessons for the next run

That is the whole loop in one picture: act, check against the record, learn from the misses.

On 40 test tickets, the agents reported success on 52.5%. The record confirmed 50.0%. One claimed success was false, which is 1 of 21. That is a silent failure rate of 4.8%.

How I tested it

I didn't want to trust a clean run, so I planted trouble. Across 40 test tickets, tools were made to fail 30% of the time, some silently. There were two customers with the same name, and refunds above the approval limit.

Some tickets carried a hidden answer key, so I could also score the checker itself. It disagreed with the key twice, both times on the same-name customers: the write landed, the check passed, and nothing confirmed it was the right person. I put that error in the README, because a checker that hides its own mistakes is asking for trust it tells you not to give an agent.

Two things went wrong along the way. I scored the agent's confidence the wrong way round, and fixing that line halved one error score (from 0.38 to 0.18). And a retry paid out a refund twice, and both passes looked clean because the end state looked right. I added a guard against repeats and a running limit per order.

I also retracted a result. A probe that looked like a perfect score turned out to come from how I generated the data, so I removed it from the README and the dashboard.

What I measured: reported against verified success, silent failures, and how well the agents' confidence matched reality (a Brier score of 0.023). The silent-failure rate of 4.8% came with a 95% confidence range of 0.008 to 0.227, which is wide because it rests on one event. What I did not measure: anything with real customers, or a run at a scale where that range would narrow.

Research

Primary research. Everything first-hand came from running the fleet. I ran an early sweep of 10 tickets on a free quota, but 4 of the 10 produced no result at all because I hit the rate limit, so I didn't trust its numbers. I then ran the 40-ticket sweep above. My notes record no user interviews.

Secondary research. Before building, I read five recent papers on false success in AI agents, and the design follows them. One found that agents claim success falsely in 45 to 48% of failures when nothing independent checks them, but only 3% when something does. Another found that AI judges barely beat chance at spotting these failures (0.65 on a 0 to 1 scale where 0.5 is a coin flip), while cheap fixed checks did far better (0.83 to 0.95). That is why my checker uses fixed rules first and an AI auditor only as a backstop. The papers are on arXiv: 2606.09863, 2607.07405, 2601.15778, 2601.06112 and 2407.18370. After the results, I also studied all 16 winners.

What happened

It did not place. Out of 12,265 registered people, 16 winners were named on 24 October 2026, and no ranks were published beyond them. The Grand Prize went to Plant Debugger. The Fortified Enterprise Fleet track went to EIR: Healthcare Agent Fleet.

Judges don't publish scores, so I can't tell you why. After the results I read all 16 winners' pages, videos and public code, and compared. Here are the gaps I think mattered.

What I learned

I picked a track whose brief named things I hadn't built. The Fortified brief listed seven Google Cloud services by name and asked for work that carries on across weeks. The winner wired all seven. I used three, and my own architecture diagram marked one of them "partial." My system also handled each ticket in one go, with nothing that could wake up days later. Next time I count the brief's nouns before I choose a track.

One event is not a result. My headline number was one silent failure out of 21. It was honest, and I published the wide uncertainty around it. But winners showed big counts or physical results: tens of thousands of rows checked, over a thousand tests, a real pull request opened. One event reads as "nothing happened."

I did not finish the small things. The rules gave extra points for a published article and a social post. I wrote drafts of both and never posted them. A draft earns nothing.

My video also opened on a number, not a person. It was a research talk. The winners opened on someone with a problem, and they ran a task live, with no cuts, before showing the numbers. My video got 20 views.

None of this is certain. All of it is cheap to fix, and it is now a checklist I use before any hackathon.

What I'd keep

Counting my own failures was the right instinct. The tracking of reported against verified success is the part I'd build again, and I'd just wrap it in a story about a person.

The live Google Cloud service is switched off, so nothing bills. The dashboard still works as a snapshot of the final run.

Links

Available · London / UK · senior AI, AI FDE, contract

Building something with AI?

Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.

or leave your email

Replies within ~24 hours · London, UK · on-site, hybrid or remote