An AI phone agent says the call went fine. I built a tool that checks. It didn't win, but it got merged.
Carson Rodrigues / November 21, 2026
6 min read • ––– views
An AI agent phones someone to confirm an appointment, comes back saying "done," and nobody knows if that is true.
| Hackathon | CALL-E "Your Code Is Calling" |
| Organiser | CALL-E (the CALLE-AI team), on Devpost |
| When | September 2026; submissions closed 14 September at 23:45 Singapore time |
| Field | Not published in my sources |
| Result and rank | Did not win. Winners were announced around 19 October; no ranks were published beyond the winners. My pull request was merged on 11 September |
| Built in | About a week: first tests against the real API on 4 and 5 September, pull request opened 5 September, video finished 9 September |
| Team | Solo |
| Money spent | Not tracked. The demo ran on Google Cloud (set to scale to zero) until I deleted it on 26 October. I did not record the cost |
| Code | In the organiser's repo |
CALL-E is a service that lets AI agents make real phone calls. Its team ran a hackathon called "Your Code Is Calling," with submissions closing on 14 September 2026 at 23:45 Singapore time. You entered by opening a pull request (a proposed change) to their public repo of phone-call apps, with a short video under three minutes. The prizes were $4,000 for Most Practical, $3,000 for Most Innovative, two $1,000 honourable mentions, and five $200 prizes for feedback on the platform.
I entered Most Practical.
What I built
When a CALL-E call ends, you get a tidy result: "task completed: true," a confidence score, and some filled-in fields like the new appointment time. Nothing in that tells you whether the person on the phone actually said those things.
Call Review Console is a review desk that sits after the call. It only reads. It never places a call.
It checks four things:
- Is each field in the result supported by what was actually said in the transcript? A time like "Thursday 2pm" has to be spoken in the call.
- Did the agent tell the person it was an AI, and was that at the start?
- If the person said "stop calling me," did the agent stop?
- How long did the agent take to answer, and were there long silences?
At the end it gives one verdict: approve, needs a human, or reject, with a one-line reason. A person can record their own decision. A benchmark page compares every call on file.
Almost all of it is plain rules, not AI. An AI model pass is there as an option, and when used it has to point to the exact line of the transcript that backs each field. I also wrote a matching skill, a set of instructions an agent can follow to do the same review without the console.
How I tested it
I wrote five made-up calls, each built to break exactly one rule: a result the transcript doesn't support, a missing AI disclosure, a "stop calling me" that was ignored, a slow agent, and a clean call that should pass. The fake phone numbers use a range reserved for fiction.
At submission there were six automated tests. The final version in the repo has 62, mostly added later for security: login checks, safe file names and the webhook. Every check is a fixed rule, and when the console can't tell, it says "needs a human" rather than guessing.
What I measured: that each of the five calls got the verdict I expected. What I did not measure: how often the console is right on real calls, or how it compares with a human reviewer. I never ran it on a real call, and it has no accuracy rate. Five fake calls prove the rules fire. They don't prove the rules are enough.
Research
Primary research. On 4 and 5 September I tested CALL-E's real API and SDK and wrote down what broke: eight items, each reproduced. For example, the call list endpoint returned an error, so a reviewer can't list their own calls. My notes record no interviews with people who run phone agents.
Secondary research. I read CALL-E's documentation and the SDK's built-in help text, which was the only reliable source for the shape of a call result because several documentation pages were missing. I also studied the repo itself, which had more than 300 merged apps. The closest one was callproof, which needs you to write a contract first. Mine needed no contract and worked on the API's own result, and I used that gap as my pitch. The review console was already on the repo's roadmap.
What happened
I opened the pull request on 5 September, early, because the repo's review queue was slow. It was merged on 11 September.
It did not win, and I can't give a rank: no field size was published where I looked, and no ranks beyond the winners. The winners were announced around 19 October and my entry wasn't among them. Being merged means the maintainers accepted the code into their repo. It does not mean it placed, and I won't pretend it does.
After the merge, the CALL-E team hardened the app, adding a login token on the console and tighter checks on the webhook, the address that receives call results.
What I learned
My first matching rule rejected a perfectly good call. The agent had said "nine a.m." and the result said "9am," and the code saw two different things. I added a step that treats times written different ways as the same. I also stopped matching yes-or-no fields by keyword at all, and mark them "needs reading" instead. A checker that wrongly rejects good calls gets switched off.
I also learned that a confidence score is the agent grading itself. Only a line in the transcript counts as evidence. In my five test calls, the score never disagreed with "task completed," so it told me nothing.
And I found out late that privacy matters at the front door. My first version hid phone numbers only when drawing the page, which left the real numbers sitting in the stored files. The later version hides them before anything is saved. I'd do that from the start now.
The hackathon also asked for feedback on the platform, and building this gave me plenty. The call list endpoint returned an error. Some docs pages were missing. Webhook messages weren't signed, so anyone could fake one. Times in the transcript had a start but no end, so I could only estimate response time and had to say so on the page. I wrote all of it up for them.
Links
- The merged pull request
- The app in the organiser's repo
- A read-only snapshot of the console (fictional calls; the buttons no longer save)
Available · London / UK · senior AI, AI FDE, contract
Building something with AI?
Voice agents, MCP servers, LLM pipelines, agentic workflows — pick a slot, drop a message, or send your email and I'll reply within a day.
Replies within ~24 hours · London, UK · on-site, hybrid or remote