I Measured How Often My AI Agents Lied About Finishing

I built this project and wrote this post for the All Things Agentic Hackathon, run by Google Cloud and Devpost.

An agent reports "refund issued." The payment gateway left it pending. The run counts as a success, the board stays green, and the customer never gets their money.

Nobody sees that one. Not the agent, not the dashboard, not the person on call at 2am.

The agent isn't lying either. That's the part that took me a while. It read a tool response that said success and believed it. It genuinely can't tell.

The number nobody publishes

Here's what bothered me. Every agent framework I looked at computes its success rate from the agent's own self-report. Which means when the agent is wrong about itself, the dashboard is wrong with it, in exactly the same direction, and you find out from a customer.

So I went looking for the rate. How often does a production agent fleet say "done" about something that didn't happen?

I couldn't find anyone publishing one. So I built the thing that measures it.

What I did

Five governed agents on a customer support workload: a controller that splits a ticket into typed tasks, a billing agent, an account agent, a vision reader for attachments, and an auditor. Each worker sees only its own task, never the whole ticket. That isolation is deliberate. It's what makes a single claim checkable on its own.

Then every claim gets checked against Firestore, which is the actual system of record. Deterministic post-conditions first. The LLM auditor only runs where no post-condition exists, because asking a model to grade a model is how you get 0.65 AUROC and a false sense of coverage.

If the agent said "done" and the record disagrees, that's a silent failure. It gets counted.

The result

40 tickets, faults injected at a 30% rate:

reported success 52.5%, verified success 50.0%
silent-failure rate 4.8%, which is 1 of 21 done-claims
95% confidence interval 0.008 to 0.227
false alarms 0%
Brier 0.023, ECE 0.033

That interval is embarrassingly wide, and I'm publishing it anyway. It's one event over 21 claims. A rate like that reads far more certain than it is, and the whole point of the project is to stop taking numbers on trust. Quoting "4.8%" without the interval would be doing the exact thing I built this to catch.

The part I nearly got wrong

My calibration was broken and it took me too long to see.

I was scoring the agent's stated confidence as if it meant P(verified = true). It doesn't. It's the agent's confidence in its own claim. So when a worker says "blocked" at 0.9, it's confidently claiming failure, and my scorer was punishing it for being right.

One line. Brier went from 0.38 to 0.18. My agents looked badly calibrated for days when the measuring instrument was the thing that was broken.

There's a lesson in there that I keep relearning: check where the number came from before you believe it, especially when the number flatters your thesis.

I also measured the checker

This is the part I'd want to see in someone else's writeup, so it goes in mine.

On those 40 runs, the verifier disagreed with hidden ground truth twice. Both times on the same ticket, where two customers share a name. The write landed, the post-condition passed, and nothing confirmed it was the right person.

A verification layer that hides its own error rate is asking for precisely the trust it tells you not to give an agent. So: 2 of 40, and here's the failure mode.

The result I threw away

I'd built an interpretability probe alongside this. A linear probe recovering the label from the model's internal activations at 1.00 AUROC, while the surface text sat at chance. It looked like a real finding.

Then I read my own data generator and found it drew confidence from the same distribution for both labels. The controls were uninformative by construction. The contrast I was proud of was an artifact of how I made the data.

I pulled it from the README, the results file, and the dashboard. It cost me a good-looking chart. Keeping it would have cost the credibility of everything standing next to it, and a retraction is cheaper than a defense.

What I'd tell you to steal

You don't need my project. You need the check.

Pick your agent's three most consequential actions. After each one, read the system of record back and compare it to what the agent claimed. Count the disagreements. Publish the count with an interval.

That's it. It's not clever. It's the reconciliation step every other critical system has kept for decades, and agent frameworks quietly dropped it somewhere between the demo and production.

In production, "done" isn't a result. It's a claim.

Code, the eval harness and the live dashboard: github.com/rodriguescarson/attest-fleet