Evals for AI Agents: Build the Suite Before You Ship

I've written before that you cannot eyeball your way to a reliable agent. This post is the practical follow-up: how I actually build eval suites for agent systems, in the order I build them, with the mistakes I made so you don't have to.

The one-sentence version: the eval suite is the spec. If a behavior matters, it's in the suite. If it's not in the suite, you have no idea whether you still have it.

Start from transcripts, not imagination

The first instinct is to sit down and write test cases from your head. Resist it. Cases you invent are the cases the agent already handles — you designed the prompt around them. The failures live in the inputs you didn't imagine.

Every eval suite I've built that was worth anything started from real transcripts. For voice agents, that meant pulling call recordings and their transcriptions; for text agents, session logs. Then a triage pass, by hand, with coffee:

Wins become golden cases: input, expected outcome, and the tool calls a correct run should make.
Failures become regression cases — the most valuable items in the suite, because they're proof the failure mode exists in your traffic.
Weird ones — the caller with two accounts, the user who switches language mid-conversation, the request that's technically two requests — become the edge-case set.

Before launch you won't have transcripts, so you bootstrap: write 20–30 cases from the customer's own examples ("here's what our users actually ask"), ship to a pilot group, then replace your invented cases with real ones within the first couple of weeks. The suite should smell like production, not like a whiteboard.

Aim for 30–50 cases before the first real user, growing toward a few hundred. Past that, curate — a bloated suite nobody reads is a different kind of vibes-based development.

Task-level and turn-level are different questions

A mistake I see constantly: teams measure turn quality ("was this response good?") and assume it composes into task success. It doesn't. An agent can produce eight individually reasonable turns and still fail to book the appointment.

So I score two layers, always:

Task completion is the metric the customer cares about. Did the refund get issued? Did the booking land in the calendar with the right time? This is usually checkable without a model — assert against the database, the API call log, the final structured output. When I can verify task completion deterministically, I do. It's cheap, fast, and never lies.

Turn-level metrics are diagnostic. Groundedness of each answer, tool-call correctness, tone, whether the agent asked a clarifying question when it should have. These tell you why task completion moved. When the task metric drops after a prompt change, turn-level scores are how you find the broken step instead of re-reading fifty traces.

The trap is optimizing turn-level scores directly. I've watched a change push "helpfulness" up while task completion fell — the agent got chattier and more reassuring, and stopped actually finishing the job. Task completion is the target; turn metrics are the debugger.

LLM-as-judge works, but only calibrated

For anything fuzzy — tone, helpfulness, whether a summary is faithful — you'll end up with an LLM judging LLM output. This works far better than its reputation, and far worse than naive usage.

What naive usage looks like: one prompt saying "rate this response 1–10," a frontier model, no reference labels, ship it. What you get is a judge that's generous, inconsistent across runs, and biased toward long, confident answers — the same biases as the model being judged.

What calibration looks like in practice:

Human-label a slice first. Take 50–100 cases and label them yourself, or better, with the customer. This is your ground truth.
Measure judge agreement against it. If the judge agrees with humans less than ~85–90% of the time on binary checks, the judge is noise. Fix the rubric before trusting a single score.
Binary or ternary beats 1–10. "Does the response contain the correct account balance: yes/no" is checkable. "Rate accuracy out of 10" invites the judge to hedge at 7.
One judge per criterion. A judge asked to score five things at once blends them. Five narrow judges with narrow rubrics agree with humans dramatically more often.
Re-calibrate when anything changes. New judge model, new rubric, new product surface — re-run the human-labeled slice. Judge drift is real and silent.

And keep spot-checking. I sample judge verdicts weekly and read them against the transcript. The day you stop auditing the judge is the day it quietly starts grading a different exam.

The eval suite is the contract

Here's the part that changed how I sell and scope this work. When you deploy an agent for a customer, the hardest conversation is "how do we know it's good enough to turn on?" Vibes don't survive that meeting. An eval suite does.

The move: turn the acceptance criteria into eval cases, together, before writing much of anything. Sit with the customer, take their top scenarios — the calls they get every day, the edge cases that burned them, the things the agent must never do — and encode each one as a case with a pass condition. Now "done" has a definition both sides signed.

This does three things at once:

Scoping gets honest. When the customer adds "oh, and it should also handle X," that's a new set of eval cases and a visible change in scope — not a vague expansion of the same milestone.
Launch gets objective. "We ship when the golden set passes at N% and the never-do set passes at 100%" is a criterion everyone can watch move week over week.
Trust survives incidents. When something breaks in production — and it will — you add the case to the suite, show the fix passing, and demonstrate it can't silently regress. That conversation goes very differently than "we tweaked the prompt, should be fine now."

At Ôdasie this became the default shape of every AI engagement: the eval suite was a deliverable, listed alongside the agent itself. Customers who couldn't read a line of code could read a pass/fail table.

Wire it into the loop or it rots

An eval suite you run "when you remember" is a suite that's three weeks stale when you need it. The mechanics that keep it alive:

Run the fast tier on every change. Deterministic assertions and cached golden cases are cheap enough to run on every PR like unit tests — the Siemens years drilled that reflex into me, and it transfers directly.
Run the full judged suite before release and on a schedule, so model-provider updates under your feet don't go unnoticed.
Every production incident ends with a new eval case. No exceptions. This is the flywheel: the suite becomes a fossil record of every way the system has ever failed, and none of those ways can come back quietly. (More on that loop in my post on agent failure modes and postmortems.)
Track the score over time, visibly. A dashboard with the trend line does more for team behavior than any policy document.

The takeaway

Build the golden set from real transcripts, not imagination. Score the task, debug with the turns. Calibrate every judge against human labels and keep auditing it. And treat the suite as the contract — with your customer, and with your future self who's about to change the prompt at 6pm on a Friday.

The teams that ship reliable agents aren't the ones with better prompts. They're the ones who can answer "did this change make things better or worse?" in ten minutes, with numbers.

Related reading