A/B Testing LLM Systems: Honest Experiments With Dishonest Variance

Offline evals tell you a change is probably better. Production tells you whether it is. The gap between those two is where A/B testing lives — and LLM systems make it harder than classic product experimentation in a few specific ways: outputs are stochastic, quality is multidimensional, and most B2B agent deployments have nowhere near the traffic the textbook assumes.

I've run these experiments on voice agents and LLM pipelines with real users on the line. Here's the version of A/B testing that survived contact with reality, organized around the questions I actually get asked.

What should you randomize?

Not everything at once. The unit of change matters more than the mechanics of the split.

Prompt changes are the most common experiment and the easiest to run — same model, same infra, just a different system prompt behind a flag. One rule: version prompts like code. Every variant gets an ID that lands in the trace, or you will be staring at a transcript in three weeks with no idea which prompt produced it.

Model changes (provider upgrade, tier downgrade, new model entirely) are the highest-stakes experiment. They shift everything at once — quality, latency, cost, refusal behavior — so pin every other variable and watch more than your primary metric. A model swap that improves answer quality but adds 800ms of first-token latency can be a net loss for a voice agent; users feel silence before they judge substance.

Retrieval and tool changes — new chunking, a reranker, a redesigned tool schema — are the sneakiest, because they change what the model sees, and the effects are input-dependent. A retrieval change can be a big win on long-tail queries and a regression on the head. Segment before you conclude.

The randomization unit should almost always be the session or user, not the request. An agent that changes personality mid-conversation because turn 3 landed in a different bucket is both a broken experience and contaminated data.

What's your guardrail set?

Every experiment gets one primary metric — task completion, resolution rate, whatever the product's honest north star is. But LLM changes fail sideways, so the guardrails matter as much as the target:

Latency — p50 and p95 time-to-first-token, and total task time.
Cost per session — a "better" prompt that doubles tool calls can quietly double your bill.
Refusal and escalation rate — a spike means the change made the model skittish or broke a tool.
Safety and never-do checks — the things your eval suite enforces at 100% shouldn't be allowed to slip in production either.
Loop and step-cap hits — a rise means the change confused the agent even if completed tasks still look fine.

The rule I hold to: a variant wins only if it beats the primary metric without breaching any guardrail. "Quality up 4%, cost up 60%" is not a win; it's a decision, and someone should make it consciously.

Do you have the traffic? Probably not.

Here's the uncomfortable arithmetic nobody wants to run. Detecting a 5-percentage-point lift on a metric sitting around 70% needs roughly 1,200–1,500 sessions per arm for a properly powered test. A deployment doing 80 sessions a day gets you there in about five weeks — per experiment.

Most agent deployments I've worked on, especially early ones, don't have that. And the failure mode isn't "we couldn't run tests." It's worse: teams run underpowered tests anyway, peek at day four, see 68% vs 74%, and ship the "winner" — which is indistinguishable from a coin flip at that sample size. An underpowered A/B test is a random number generator with a dashboard.

The honest playbook at low traffic:

Do the power math before you start. If the answer is "eight weeks," say eight weeks — or accept that you're only powered to detect huge effects, and only act on huge effects.
Prefer paired comparisons. Run both variants on the same inputs (replayed traffic offline, or interleaving online) so between-user variance stops eating your signal. Paired designs need far fewer samples for the same confidence.
Lean harder on offline evals — more on when that's enough below.
Don't peek-and-ship. If you must monitor continuously, use a sequential method honestly; otherwise fix the duration and keep your hands off.

Interleaving: the underrated middle path

For ranking and retrieval changes there's a trick worth stealing from the search world: instead of splitting users between system A and system B, blend both systems' candidates into a single result set and see which system's contributions the user actually clicks or uses. Sensitivity is typically an order of magnitude better than a traditional A/B split, which is exactly what a low-traffic deployment needs.

For agents, the same instinct applies in adapted forms. Run the new retrieval pipeline in shadow on live traffic and compare what it would have fed the model. Generate candidate responses from both prompts and have a calibrated judge pick pairwise winners on the same real inputs. None of this is a user-facing experiment, but it's paired, high-sensitivity evidence from production traffic — often the best evidence you can afford.

When are offline evals enough?

More often than the experimentation literature would like. My working rule:

Offline is enough when the change is a refactor that should be behavior-neutral and the golden set confirms it; when the eval suite directly covers the behavior being changed and the effect size offline is large; or when the change fixes a specific failure mode you have regression cases for. Shipping a fix for a known bug does not need a three-week experiment — it needs the regression case to pass and production monitoring to confirm.

Production testing is non-negotiable when the change touches latency or cost in ways users feel; when you're swapping models (offline evals never fully capture a model's production personality); when the behavior depends on real user reactions — do people actually answer the new clarifying question, or hang up?; and when the metric that matters is downstream of the transcript, like conversion or repeat usage, which no judge can score.

A useful mental model: offline evals test the system's behavior; online experiments test the users' behavior. When the question is about your system, offline is cheap and fast. When the question is about humans, only humans can answer it.

The loop that makes both work

The teams that get this right don't treat offline and online as rival methodologies. They run a loop: eval suite gates the change, a flagged rollout at 5–10% catches what the suite missed, guardrail metrics decide whether it expands, and whatever production surprises you becomes a new eval case so the next change is tested against it offline — where it's cheap. Over time the suite absorbs more of the risk and the experiments get reserved for the questions only users can answer.

That's the real endgame: not more A/B tests, but fewer — because your offline evals have earned the right to be trusted.

The takeaway

Randomize at the session level and one variable at a time. Guard the metrics that fail sideways — latency, cost, refusals — not just the one you're optimizing. Do the power math and respect what it tells you; at low traffic, paired and interleaved designs beat naive splits by an order of magnitude. And let offline evals carry everything they can, so production experiments are spent on the only thing they're uniquely good at: finding out what real people actually do.

Related reading