My agents said 'done' when it wasn't. I built a fleet that counts those lies. It didn't place.
Carson Rodrigues / November 21, 2026
7 min read • ––– views
An AI agent tells a customer "your refund is issued." The payment system never sent it, nobody checks, and the dashboard stays green.
| Hackathon | All Things Agentic, Fortified Enterprise Fleet track |
| Organiser | Google, on Devpost |
| When | 3 to 31 August 2026 (I submitted on 30 August) |
| Field | 12,265 registered |
| Result and rank | Did not place. 16 winners were named; no ranks were published beyond the winners |
| Built in | About 5 days (first work 25 August, submitted 30 August) |
| Team | Solo |
| Money spent | Not tracked. My early test runs used Google's free tier; I did not record cloud or model costs. Everything on Google Cloud was deleted on 25 October, so nothing bills now |
| Code | github.com/rodriguescarson/attest-fleet |
That was the problem I built for in Google's All Things Agentic hackathon, which ran on Devpost from 3 to 31 August 2026. It had 12,265 registered people and three tracks. I entered the Fortified Enterprise Fleet track, the one about running many agents inside a company without losing control of them. I submitted on 30 August.
Judges scored three things: how useful the agent was and whether it acted on its own (40%), how well it was built on Google's tools (30%), and the demo and how ready it was for real use (30%).
What I built
Attest Fleet is a small customer-support team made of five AI agents. One reads a ticket and splits it into tasks. One handles billing. One handles accounts. One reads photos that customers attach. A fifth, running on a different model family, acts as an auditor.
The idea is simple. When a worker says "done," I treat that as a claim, not a fact. A separate checker then looks at the real database and asks whether the world actually changed.
If the agent said "done" and the record disagrees, that is a silent failure. I count it and publish the number.
A few more rules sit around that. A refund over a set limit stops and waits for a human. There is a kill switch that stops the whole fleet. Each run keeps an evidence trail, so you can see exactly where a claim and the record split apart.
That is the whole loop in one picture: act, check against the record, learn from the misses.
On 40 test tickets, the agents reported success on 52.5%. The record confirmed 50.0%. One claimed success was false, which is 1 of 21. That is a silent failure rate of 4.8%.
How I tested it
I didn't want to trust a clean run, so I planted trouble. Across 40 test tickets, tools were made to fail 30% of the time, some silently. There were two customers with the same name, and refunds above the approval limit.
Some tickets carried a hidden answer key, so I could also score the checker itself. It disagreed with the key twice, both times on the same-name customers: the write landed, the check passed, and nothing confirmed it was the right person. I put that error in the README, because a checker that hides its own mistakes is asking for trust it tells you not to give an agent.
Two things went wrong along the way. I scored the agent's confidence the wrong way round, and fixing that line halved one error score (from 0.38 to 0.18). And a retry paid out a refund twice, and both passes looked clean because the end state looked right. I added a guard against repeats and a running limit per order.
I also retracted a result. A probe that looked like a perfect score turned out to come from how I generated the data, so I removed it from the README and the dashboard.
What I measured: reported against verified success, silent failures, and how well the agents' confidence matched reality (a Brier score of 0.023). The silent-failure rate of 4.8% came with a 95% confidence range of 0.008 to 0.227, which is wide because it rests on one event. What I did not measure: anything with real customers, or a run at a scale where that range would narrow.
Research
Primary research. Everything first-hand came from running the fleet. I ran an early sweep of 10 tickets on a free quota, but 4 of the 10 produced no result at all because I hit the rate limit, so I didn't trust its numbers. I then ran the 40-ticket sweep above. My notes record no user interviews.
Secondary research. Before building, I read five recent papers on false success in AI agents, and the design follows them. One found that agents claim success falsely in 45 to 48% of failures when nothing independent checks them, but only 3% when something does. Another found that AI judges barely beat chance at spotting these failures (0.65 on a 0 to 1 scale where 0.5 is a coin flip), while cheap fixed checks did far better (0.83 to 0.95). That is why my checker uses fixed rules first and an AI auditor only as a backstop. The papers are on arXiv: 2606.09863, 2607.07405, 2601.15778, 2601.06112 and 2407.18370. After the results, I also studied all 16 winners.
What happened
It did not place. Out of 12,265 registered people, 16 winners were named on 24 October 2026, and no ranks were published beyond them. The Grand Prize went to Plant Debugger. The Fortified Enterprise Fleet track went to EIR: Healthcare Agent Fleet.
Judges don't publish scores, so I can't tell you why. After the results I read all 16 winners' pages, videos and public code, and compared. Here are the gaps I think mattered.
What I learned
I picked a track whose brief named things I hadn't built. The Fortified brief listed seven Google Cloud services by name and asked for work that carries on across weeks. The winner wired all seven. I used three, and my own architecture diagram marked one of them "partial." My system also handled each ticket in one go, with nothing that could wake up days later. Next time I count the brief's nouns before I choose a track.
One event is not a result. My headline number was one silent failure out of 21. It was honest, and I published the wide uncertainty around it. But winners showed big counts or physical results: tens of thousands of rows checked, over a thousand tests, a real pull request opened. One event reads as "nothing happened."
I did not finish the small things. The rules gave extra points for a published article and a social post. I wrote drafts of both and never posted them. A draft earns nothing.
My video also opened on a number, not a person. It was a research talk. The winners opened on someone with a problem, and they ran a task live, with no cuts, before showing the numbers. My video got 20 views.
None of this is certain. All of it is cheap to fix, and it is now a checklist I use before any hackathon.
What I'd keep
Counting my own failures was the right instinct. The tracking of reported against verified success is the part I'd build again, and I'd just wrap it in a story about a person.
The live Google Cloud service is switched off, so nothing bills. The dashboard still works as a snapshot of the final run.
Links
- Code on GitHub
- Snapshot of the dashboard (read-only; the buttons no longer act)
Available · London / UK · senior AI, AI FDE, contract
Building something with AI?
Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.
Replies within ~24 hours · London, UK · on-site, hybrid or remote