Observability for LLM and Agent Systems: See What Your Agent Actually Did

The first production incident on any agent system goes the same way. A user reports a wrong answer, someone asks "what did the agent actually do?", and the room goes quiet — because the logs show an HTTP 200 from the model provider and nothing else. The prompt that was sent, the tools that ran, the context that was retrieved: gone.

Traditional observability tells you the service was up. Agent observability has to tell you why the system said what it said — and those are very different instruments. After building this out across voice agents and LLM pipelines serving real traffic, here's the setup I now consider table stakes.

The trace is the unit of truth

An agent request isn't one call; it's a tree. One user turn fans out into a planning call, three tool executions, a retrieval query, maybe a sub-agent, a summarization pass, and a final generation. When something goes wrong, the bug is in one node of that tree, and without tracing you're reconstructing it from fragments across four log streams.

So the foundation is boring and non-negotiable: one trace ID per user session, one span per model call and per tool call, propagated everywhere — including across service boundaries and into async workers. OpenTelemetry's GenAI conventions have made this reasonably standard; whether you use a dedicated LLM observability product or your existing tracing stack matters less than actually doing it.

What each span needs to capture, beyond the usual timing:

Model spans: model ID, prompt version ID, token counts in and out, finish reason, whether the cache hit, and the full request/response payload (with the redaction caveats below).
Tool spans: tool name, arguments as the model sent them, the result — and critically, an explicit distinction between "succeeded with empty result" and "failed." Silent tool failures are the most common agent bug I see, and a trace that conflates those two states can't reveal them.
Retrieval spans: the query, the chunks returned, their scores. Half of all "hallucination" reports turn out to be retrieval handing the model the wrong context; you can't see that without logging what was retrieved.

The payoff test is simple: given a complaint like "the agent told me my order shipped and it hadn't," can one engineer go from user report to the exact span where things went wrong in under five minutes? If the answer is no, the incident review will be archaeology.

Logging prompts and completions without stepping on a rake

Here's the tension: the prompt and completion are the most valuable debugging artifacts you have, and they're also where the PII lives. Users type phone numbers, addresses, health details, and account numbers into chat boxes and say them to voice agents. If you log payloads naively, your observability stack quietly becomes an unaudited copy of your most sensitive data — searchable by anyone with dashboard access, retained forever.

The practices that keep you out of trouble:

Redact at the edge, before ingestion. Run detection for emails, phone numbers, card and account numbers, and names on both prompts and completions before they hit the log pipeline. Redaction after storage is not redaction.
Tokenize, don't just delete. Replace values with stable placeholders (<PHONE_1>, <ACCOUNT_2>) so the trace remains debuggable — you can still see the agent used the wrong placeholder without seeing the number itself.
Tier your retention. Full redacted payloads for a short debugging window (days to a few weeks), metadata and metrics for the long term. Most debugging happens within 72 hours of the request; design for that.
Gate raw access. If a regulated customer requires unredacted logging for audit, that store gets separate access control and its own audit log. Dashboard access and raw-payload access are different privileges.
Put logging in the data processing agreement. For enterprise deployments, what you log, where it lives, and how long it lives is a contract term, not an engineering afterthought. I've had deals where log retention got more scrutiny than model choice.

The dashboards that actually get looked at

I've built elaborate dashboards nobody opened and four-panel ones checked every morning. The difference is whether each panel answers a question someone actually has. The ones that earn their place:

Cost, per session and per outcome. Total daily spend is vanity; cost per completed task is the number that matters, and its distribution matters more than its mean. A healthy agent has a tight cluster and a short tail. A growing tail of expensive sessions means loops, bloated context, or a tool that's forcing retries. Break spend down by prompt version and model so a regression shows up attributed, not as an unexplained bump in the invoice.

Latency, at the percentiles users feel. For interactive and especially voice systems, p95 time-to-first-token is the product-quality metric. Track it per model and per pipeline stage — when it degrades, you want to know immediately whether it's the provider, your retrieval step, or a prompt that grew 3,000 tokens last Tuesday.

Behavioral counters. Tool-call error rate per tool, step-cap hits, refusal rate, escalation-to-human rate, empty-retrieval rate. These are the agent's vital signs, and they move before task-completion metrics do.

Alerting: drift doesn't page you, so make it

Infrastructure fails loudly; model behavior fails quietly. The provider updates a model, a data source changes shape, users start asking about something new — and nothing errors. The system keeps returning 200s while quality slides. If you only alert on errors, your users become your drift detection.

What I alert on instead:

Refusal spikes. Refusal rate jumping from its baseline is the single highest-signal behavioral alert. It means a model update changed dispositions, a prompt change backfired, or something upstream is mangling input. It's cheap to compute — classify completions for refusal patterns — and it catches provider-side changes hours before anyone files a ticket.
Distribution shifts. Response length, tool-call frequency per session, token usage per turn. None of these need an LLM to compute, and a sustained shift in any of them means the system's behavior changed, whether or not anyone meant it to.
Judge-scored canaries. A small fixed set of golden inputs run on a schedule against production infrastructure, scored by the same calibrated judges as the eval suite. When a canary that passed for six weeks starts failing, something under you moved.
Cost velocity. Spend rate per hour with a hard threshold. A runaway loop at 2am should page someone before it becomes a line item.

Tune these to page rarely. A behavioral alert that cries wolf weekly gets muted, and a muted alert is worse than none because everyone believes coverage exists.

The flywheel, again

The reason observability deserves this investment isn't dashboards for their own sake. Traces are where your next eval cases come from, where postmortems get their facts, and where A/B test guardrails get their data. Every improvement loop in an agent system starts with the question "what actually happened?" — observability is just the discipline of being able to answer it.

Build it before launch. The week you need it is the week it's too late to add.

Related reading