CI/CD for Prompts and Agent Configs: Ship Prompt Changes Like Code, Roll Back in Seconds

Here's a failure mode I've now seen enough times to call it a pattern: a team with excellent CI/CD for their application code — tests, reviews, staged rollouts, the works — editing their production system prompt in a web console. Someone "tightens the wording" on a Thursday afternoon. Friday morning, escalations are up 30% and nobody knows why, because the change isn't in git, wasn't reviewed, wasn't evaluated, and can't be diffed.

Prompts and agent configs are the highest-churn, highest-blast-radius artifacts in an AI system, and most teams manage them with the least discipline. At Ôdasie, where I led a team shipping multiple production AI systems, putting prompts through the same pipeline as code was one of the changes that took our release cycles from weeks to hours — because it made prompt changes safe enough to ship constantly.

Here's the pipeline, piece by piece.

Prompts are versioned artifacts, not database rows

Everything that shapes agent behavior lives in the repo:

agents/
  support-agent/
    prompt.md              # system prompt, markdown, reviewable
    tools.json             # tool definitions and schemas
    config.yaml            # model, temperature, budgets, routing
    evals/
      cases.jsonl          # eval cases for this agent
  overlays/
    customer-a.yaml        # per-customer deltas (more below)

This buys you everything git buys code: diffs (you can see exactly which sentence changed), blame (you know who changed it and why, from the PR), review (a second pair of eyes on the artifact most likely to alter production behavior), and atomic versioning — the prompt, tools, and config that were tested together ship together, as one immutable release like support-agent@2026-03-24.3.

That last point matters more than it looks. A prompt that says "use the refund tool for amounts under the limit" is coupled to the tool schema and the config value it references. Version them separately and you will eventually run a new prompt against old tools. Version them as a unit and that whole failure class disappears.

The rule with no exceptions: nobody edits behavior in production. Not during incidents, not for "one quick fix." The console is read-only; the repo is the source of truth.

Eval gates: the tests of the prompt world

Code changes get unit tests; prompt changes get evals. Same slot in the pipeline, same authority to block a merge:

# .github/workflows/agent-ci.yml (abridged)
on:
  pull_request:
    paths: ["agents/**"]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pnpm evals:run --agent support-agent --suite regression
      - run: pnpm evals:report --fail-below 0.92 --budget-check

What goes in the suite, in order of value:

Regression cases from production failures. Every incident and every bad transcript found in hypercare becomes an eval case. This is the compounding asset — six months in, your suite encodes everything the system has ever gotten wrong.
Golden paths. The core workflows, asserted end-to-end: right tool called, right arguments, escalation triggered when it should be.
Guardrail probes. Injection attempts, out-of-scope requests, tone traps. These must keep failing safely after every change.
Budget checks. Token counts per case, so a prompt edit that doubles input cost fails CI before it hits the invoice (why that matters).

Two hard-won practicalities. First, evals are noisy — LLM-judged scores wobble — so gate on aggregates over the suite with a small tolerance band, not on single cases, or your team will learn to ignore red builds. Second, keep the suite fast enough to run on every PR (minutes, not hours); a slow gate becomes a skipped gate. A small model as judge for most cases keeps it quick and cheap.

The cultural effect is the real payoff: when the gate exists, non-engineers can safely propose prompt changes. The pipeline, not seniority, decides what ships.

Canary rollouts: evals lie, traffic doesn't

A green eval suite means the change probably isn't broken. It does not mean the change is better — eval suites cover what you thought to test, and production users are more creative than that. So prompt changes ship the way risky code ships: to a slice first.

The mechanics are simple if your runtime resolves the agent version per request:

Ship support-agent@N+1 alongside N, route 5% of traffic to it (hash on conversation ID so a session never flips versions mid-conversation).
Watch the canary's metrics against baseline: escalation rate, tool-failure rate, latency, tokens per conversation, thumbs-down rate.
Hold for a meaningful sample — for prompts that's usually hours, not days — then ramp 5% → 25% → 100%, or kill it.

Canaries catch a class of failure evals structurally can't: the prompt tweak that's fine in isolation but subtly changes tone across long conversations, or shifts the tool-call distribution in a way that only shows up against real data. In a multi-tenant setup, canary on your friendliest tenant first — more on that here.

Per-customer config layering

Enterprise deployments diverge per customer — different tone, different escalation rules, different tools enabled. The trap is forking the prompt per customer; three months later you have nine prompts drifting independently and no way to ship an improvement to all of them.

The structure that survives: one base, thin overlays.

# agents/overlays/customer-a.yaml
extends: support-agent@2026-03-24.3
model_tier: premium
tools:
  disable: [refund_tool]        # customer A handles refunds manually
prompt_fragments:
  tone: "formal-en-GB"          # named fragment, from a curated library
escalation:
  handoff_target: "customer-a-tier2-queue"

The rules that keep it sane:

Overlays contain data, not prose. Flags, thresholds, routing targets, and references to named, tested prompt fragments — never free-text prompt additions. Free-text overlays are unreviewable forks wearing a config costume.
The base moves for everyone. Improvements and fixes land in the base and reach every customer on their next rollout. If a customer "needs" a base change that others can't take, that's a product decision to surface, not a fork to hide.
CI runs evals per overlay. The matrix is base × overlays; a base change that breaks customer A's configuration fails in CI, not in customer A's production.

Rollback in seconds — because you versioned

This is where the whole design pays off. If a release is an immutable, atomic artifact and the runtime resolves a pointer per agent per environment, then rollback is moving the pointer:

agentctl rollback support-agent --env prod --to 2026-03-24.2
# pointer flipped; next request serves the previous version

No build, no deploy, no container restart — the artifacts are data, both versions already exist, and the switch is one config write. Seconds, not the tens of minutes a code rollback takes.

For agent systems this speed is not a luxury. A bad prompt doesn't throw a stack trace; it converses wrongly, politely, at full production volume, in front of your customer's customers. The window between "detected" and "reverted" is measured in damaged conversations. When rollback is instant and rehearsed (I rehearse it in staging as part of every go-live checklist), the on-call decision becomes easy: roll back first, diagnose second.

The takeaway

Treat prompts and agent configs as what they are: the most frequently changed, most behavior-critical artifacts in your system. Version them atomically in git, gate them with evals in CI, ship them through canaries, layer per-customer differences as data on a single base, and make rollback a pointer flip. None of this is exotic — it's the CI/CD playbook you already run for code, applied to the artifacts that actually change every week. The teams that do this ship prompt improvements daily with confidence. The teams that don't edit production in a web console and find out on Friday.

Related reading