I built Attest to catch AI agents that say they're done when they aren't
Carson Rodrigues / November 21, 2026
6 min read • ––– views
If your AI agent tells you it fixed the bug, how do you know it did?
Most dashboards just add up what the agent says about itself. If the agent says "done" and the tests are still red, that run still counts as a success.
| Hackathon | Ready, Spec, Ship |
| Organiser | Kiro |
| When | 8 to 23 August 2026 |
| Field | Not published |
| Result and rank | No result ever announced |
| Built in | Under five days: first commit 19 August, submitted 23 August |
| Team | Solo |
| Money spent | Nothing paid. Kiro's Free tier, no paid plan, and Attest has no running costs (the README says so) |
| Code | github.com/rodriguescarson/attest |
The hackathon
Kiro is an AI coding tool built around specs: you write down what you want, it turns that into a design and a task list, and only then writes code. Its hackathon, Ready, Spec, Ship, ran from 8 to 23 August 2026.
Judges scored 100 points. Forty went to how complete and useful the app was, 20 to how well you used Kiro, 20 to documentation, 15 to originality, and 5 to the demo video. You had to publish your code with Kiro's project folder in it, so judges could read the trail of specs for themselves.
What I built
Attest takes a log of agent runs and tells you how far the agent's story is from what happened.
Each run in the log says whether the agent claimed success. It also says whether a set of independent checks passed, such as "the tests pass" and "nothing else broke". A run only counts as truly successful if every required check passes.
Then Attest reports the gap. The headline number is the silent failure rate: the share of runs where the agent said it worked and the checks say it didn't. It also reports the opposite, false alarms, where the agent said it failed and it was fine.
It also looks at confidence. If an agent says it's 95 percent sure, is it right about 95 percent of the time? And it suggests a cutoff: below this confidence, send the run to a human, and here's how many runs that sends you.
On the ten made-up runs in the demo, the agent claims 70 percent success and the checks confirm 60 percent. One run in five is a silent failure. The dashboard labels that data as synthetic, in red, at the top.
You can use it from the command line or in a browser dashboard. Everything runs on your own machine. No account, no keys, no network.
I built it with Kiro's spec workflow. I wrote three short guidance files by hand first, saying what a silent failure is and what rules the code must follow. Then three specs, one each for the scoring engine, the command line tool and the dashboard, with 32 tasks in total. Kiro wrote the code. A hook kept the metrics documentation in step with the code, and was forbidden from touching the hand-calculated test values.
Research
Primary research. The idea and the metric definitions come from my own earlier research on whether agents' reports about themselves can be trusted. I didn't interview anyone or have anyone else test Attest, and I had no real agent logs to run it on.
Secondary research. I read Kiro's official rules and scoring on 19 August and built to the rubric. I followed Kiro's own advice to write one spec per feature. The calibration measures are the standard ones. I didn't study other entries.
How I tested it
There are 150 automated tests: 72 for the scoring engine, 53 for the command line tool and 25 for the dashboard.
The key move was writing the right answers first. I worked out every metric by hand from the ten-run example, and had those values checked, before any code existed. The tests compare the code to those hand-made answers, to a billionth of a point, so the code can't just agree with itself. That covers the headline rates, all ten points on the confidence curve, and the split for out-of-distribution runs (the ones the agent wasn't built for), 0.500 against 0.1667.
I added guards for the sneaky cases. Empty confidence bins are skipped. The curve is asserted to stay lumpy, not smoothed. And at 100 percent coverage it must equal the verified success rate exactly. A bad log must fail loudly, with the line, run and field named.
Before submitting I cloned the public repo cold. Install, tests, build and the demo all passed.
What I didn't measure: Attest has only been scored on made-up runs, so I can't say how it behaves on a real agent's logs.
What happened
I submitted it before the deadline. Kiro never published winners. No result was ever announced, so there is no rank and no field size to report, and I can't tell you how it did or why.
What I learned
Pin the numbers before you write the code. Four of the metric definitions are easy to get subtly wrong, so I put worked answers into the specs as acceptance criteria, calculated by hand. Every metric came out right on the first run. The only fix the whole build needed was a file-import quirk, not maths.
A written rule isn't a locked door. My guidance file said the dashboard would use Next.js. Kiro built it with Vite, a lighter tool, which was the better choice because the dashboard has to work offline. But it didn't tell me it had changed course, and I left the guidance file contradicting the code. If the spec is the source of truth, update it in the same change.
Say how the work was really done. The README says I was away from the machine for part of the build, and that an AI assistant sent some of the prompts to Kiro for me. The specs, the metric definitions, the review and the decisions were mine. Kiro wrote all the implementation either way. Judges asked for "meaningful human direction", and I'd rather show where it was than imply it was everywhere.
The ten-run demo is tiny. It proves the tool works, not that your agent is as bad as mine pretended to be. Point Attest at a few hundred real runs before you believe a number from it.
Links
- Code and docs (public): github.com/rodriguescarson/attest
- Try it: clone the repo and follow the README to score the built-in demo. It needs no keys and no account.
Available · London / UK · senior AI, AI FDE, contract
Building something with AI?
Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.
Replies within ~24 hours · London, UK · on-site, hybrid or remote