Cost Engineering for LLM Workloads: From Token Budgets to a Number Your Customer Understands

Every LLM deployment has the same arc: nobody thinks about cost during the pilot, then usage grows, then someone opens the invoice and suddenly cost is the topic of the next steering call. I'd rather skip to the end: treat cost as an engineering requirement from day one, the same way you treat latency.

The good news is that LLM cost is unusually optimizable. Unlike most cloud spend, it's driven by decisions you make per request — what you put in the prompt, which model you call, what you cache. I've repeatedly seen teams cut spend by well over half without touching product quality, purely by working through the ladder below in order.

That order matters. Each rung is roughly increasing in effort, and the cheap rungs often make the expensive ones unnecessary.

Rung 1: Give every request a token budget

You cannot optimize what has no target. So before any clever tricks, set a token budget per request type — a number, written down, enforced in code:

// budget per request type, checked in CI and alerted on in prod
const TOKEN_BUDGETS = {
  "classify-intent":    { in: 800,   out: 50 },
  "answer-with-rag":    { in: 6_000, out: 700 },
  "agent-tool-loop":    { in: 40_000, out: 4_000 }, // total across turns
} as const;

The budgets are illustrative — yours come from measuring real traffic — but the discipline isn't. Once budgets exist, three things happen:

Prompt bloat becomes visible. The system prompt that grew from 400 to 3,000 tokens over six months of "just add a rule for this edge case" now trips an alert instead of silently taxing every single request.
Retrieval gets honest. Stuffing twelve chunks into context "to be safe" blows the budget; you're forced to measure whether chunks five through twelve ever change the answer. (Usually: no.)
Regressions get caught in CI. A change that doubles input tokens fails the build the same way a change that doubles latency should. I wire this into the same eval pipeline that gates prompt changes.

Log tokens in and out on every request, tagged by request type and customer, from the first day. This is the single highest-leverage cost decision you'll make, because every other rung depends on this visibility.

Rung 2: Prompt caching — the discount you're probably leaving on the table

Most providers now offer prompt caching: mark the stable prefix of your prompt as cacheable and pay a fraction of the input price on cache hits. For agent workloads this is enormous, because agents are built out of stable prefixes — the system prompt, the tool definitions, the retrieved knowledge — with only a thin layer of conversation changing per turn.

Getting the hits requires structuring prompts deliberately:

Stable content first, volatile content last. System prompt, tool schemas, and reference documents at the top; the user's message and conversation state at the bottom. One dynamic value spliced into the middle of the prefix — a timestamp, a request ID — and your hit rate quietly collapses.
Cache breakpoints at real boundaries. After the system prompt and tool definitions; after long reference material shared across a session.
Watch the hit rate like you watch error rate. It's a metric that degrades silently. A refactor that reorders prompt assembly can double your effective input cost with zero functional change, and nothing but the invoice will tell you — unless you dashboarded it.

In multi-turn agent loops, where every turn resends the whole transcript, caching routinely takes the largest line item on the bill and shrinks it several-fold. It's an afternoon of work. Do it before anything fancier.

Rung 3: Model routing — small model first

The most expensive habit in LLM engineering is sending every request to your best model. Traffic is never uniform: a large share of real-world requests are simple — a classification, a short factual lookup, a rephrasing — and a small model handles them indistinguishably well at a fraction of the price.

The pattern I deploy:

Route by task type first. Intent classification, extraction, formatting, summarization of short text → small model, always. These are solved problems at the small-model tier.
Escalate on signal, not vibes. The small model attempts the request; escalate to the large model when confidence is low, when the task matches known-hard patterns, or when the first attempt fails validation. Define the escalation triggers explicitly and log every escalation.
Let evals police the boundary. Run your eval suite per-route. If small-model quality on a route drifts below threshold, the fix is moving that route up a tier — a config change — not a rewrite.

Two honest caveats. Routing adds a failure mode — a misroute is a quality bug users feel — so start conservative and expand small-model coverage as evals prove it out. And measure end-to-end: an escalation path that runs both models on 40% of traffic can cost more than just calling the big model. The logs from Rung 1 tell you which side of that line you're on.

Rung 4: Batch what nobody is waiting for

Interactive requests need to be fast. But a surprising amount of LLM work isn't interactive — nightly document classification, embedding refreshes, report generation, eval runs, backfills. Most providers offer batch APIs at roughly half price with relaxed latency guarantees measured in hours.

The engineering move is architectural: separate the interactive path from the deferrable path early. Queue deferrable work (SQS and a worker fleet is my usual shape — same pattern as in my AWS deployment guide), flush it through the batch API, and reserve the synchronous, full-price path for requests a human is actively waiting on. Teams that never draw this line pay interactive prices for background jobs forever, and it never shows up as a bug — only as margin.

Rung 5: Cache whole answers

The cheapest LLM call is the one you don't make. If your workload has repeated questions — and support, internal help desks, and documentation assistants absolutely do — cache complete responses:

Exact-match first. Normalize the input (case, whitespace, tenant-specific noise) and hash it. Trivial to build, zero quality risk, and on FAQ-heavy traffic the hit rate is better than you'd guess.
Semantic caching second, carefully. Embedding-similarity lookups catch paraphrases, but a false hit serves someone a confidently wrong answer. Set thresholds conservatively, and only serve semantic hits for answers that aren't user- or state-specific.
Invalidate on knowledge changes. A cached answer built on last month's policy document is a liability, not a saving. Tie cache invalidation to your knowledge-base update pipeline, and scope every cache entry to the tenant — cross-tenant answer leakage is a security incident, not a cost optimization (more on that here).

The number that ends the pricing argument: cost per conversation

Here's the forward-deployed part. Enterprise customers don't think in tokens, and every hour a steering meeting spends puzzling over token line items is an hour of eroding confidence. What lands is one number on one slide:

"A conversation with the agent costs about $0.11. The process it replaces costs you about $6.50 in handle time." (Illustrative numbers — but that's the shape of the sentence.)

Cost per conversation is just your total model spend divided by completed conversations, tagged by customer and use case — trivial to compute if you built the per-request logging in Rung 1. Report it every month, alongside its trend. It reframes every conversation you'll have:

The invoice stops being scary, because it's anchored to delivered value per unit.
Optimization work becomes legible: "we cut cost per conversation 38% this quarter" is a sentence a sponsor repeats to their CFO.
Pricing and capacity discussions get grounded: everyone can multiply one number by projected volume.

I've watched this single metric defuse cost escalations that token-level dashboards only inflamed.

The takeaway

LLM cost yields to ordinary engineering discipline: budget tokens per request and log everything; structure prompts for caching; route to small models by default and escalate on evidence; batch the work nobody's waiting for; cache answers you've already produced. Then roll it all up into cost per conversation — the number that turns your bill from a threat into a business case. Do the rungs in order. The boring ones pay for the clever ones.

Related reading