Data Pipelines for AI Agents: Feeding the Model Without Poisoning It

Most conversations about agent quality fixate on the model and the prompt. In my experience deploying agents into real businesses, the ceiling is set earlier — by the pipeline that feeds them. An agent answering from a knowledge base that's three weeks stale, missing half the product catalog, or quietly carrying customers' phone numbers into every context window isn't a prompt problem. It's a data engineering problem wearing an AI costume.

Having run backends where data flowed in from tens of thousands of locations, I can tell you the pipeline work is unglamorous and decisive. Here's how I structure it for agent systems — following the data from source to context window.

Start with a source inventory, not an architecture

Before choosing tools, list every source the agent needs and answer four questions per source: How do I read it? How do I know when it changed? How fresh does the agent's view need to be? What's in it that must never reach the model?

A typical deployment mixes wildly different answers — a help-center CMS with webhooks, a product database you can query directly, a shared drive of PDFs nobody owns, a ticketing system with an API but brutal rate limits. The pipeline design falls out of this table. Skipping it is how you end up with one heroic nightly job trying to treat a webhook-rich SaaS API and a folder of scanned PDFs the same way.

Also decide, per source, whether the agent needs retrieval over a synced copy (docs, policies, catalogs → embed and index) or live lookup at request time (order status, account balance → call the API through a tool; see the adapter patterns post). Sync what's referenced; fetch live what's stateful. Syncing account balances into a vector store is a bug you ship on purpose.

ETL vs. event streams: pick per source, not per religion

The batch-versus-streaming debate mostly dissolves when you frame it per source:

Batch ETL (a scheduled job that extracts, transforms, loads) wins when the source has no change notifications, changes slowly, or requires expensive processing — document parsing, chunking, embedding. It's also the only honest option for the PDF-folder class of sources.
Event streams (webhooks, CDC, queues) win when the source announces changes and freshness matters — the help article that was just corrected, the price that just changed. The mechanics are the standard event-driven playbook: verify, enqueue, dedupe, DLQ (full walkthrough here).
The hybrid is the honest default: events for low-latency updates, plus a periodic reconciliation sweep that re-crawls the source and heals whatever the events missed. Events keep you fresh; the sweep keeps you complete. You need both, because webhook delivery is at-least-once but not always-ever.

One agent-specific wrinkle: an "update" to a document usually means re-chunk, re-embed, and replace — and delete the old chunks. Half-updated documents, where new chunks coexist with stale ones, produce the most confusing retrieval bugs I've debugged. Make document replacement atomic from the index's point of view.

Scrub PII at ingestion, not at retrieval

This is the strongest opinion in this post: the right place to remove sensitive data is the pipeline's front door. If PII makes it into your vector store, it will eventually make it into a context window, and from there into a model provider's logs, an eval trace, a debugging screenshot. Scrubbing at retrieval time means betting your compliance story on a filter that runs on every request and must never miss. Scrubbing at ingestion means the sensitive data simply isn't in the system.

In practice, the ingestion stage runs a scrubbing pass per record: pattern-based detection for the structured stuff (emails, phone numbers, card-like numbers, national IDs), an NER pass for names where the use case demands it, and pseudonymization over deletion where the pipeline needs continuity — replace the value with a stable token (customer_4821) so the agent can still reason about "the same customer" without knowing who they are. A reversible mapping, if you need one, lives in a separate access-controlled store, never in the index.

Two habits that keep this honest: log counts of what was scrubbed per batch (a sudden drop to zero means your scrubber broke, not that the data got clean), and add eval cases that try to retrieve known-planted synthetic PII. If the canary comes back, the pipeline failed — better you find out than an auditor.

Freshness SLAs: staleness is a product decision

"How fresh is the agent's knowledge?" deserves a numeric answer per source, agreed with the customer, not a shrug. I write them down like this — illustrative numbers, but the shape is real:

| Source | Freshness SLA | Mechanism | | --- | --- | --- | | Help-center articles | 15 minutes | webhook + hourly sweep | | Product catalog | 5 minutes | CDC stream | | Policy PDFs | 24 hours | nightly ETL | | Order status | live | tool call, never synced |

Then make the pipeline measure itself: every synced record carries a synced_at, and a monitor tracks the lag between source-change and index-update per source. Alert when a source blows its SLA — a silently stalled sync is the worst failure mode in this whole post, because the agent keeps answering, confidently, from a snapshot of the past.

And surface staleness to the agent. Passing "knowledge from source X last updated 26 hours ago" in context lets the model hedge ("as of yesterday…") or fall back to a live lookup. Models handle known-stale data gracefully; they can't handle staleness you hide from them.

Backfills: design them on day one

Every pipeline eventually needs to reprocess everything — a better chunking strategy, a new embedding model, a scrubber bug fixed, a customer who shows up with five years of history on day one. Teams that treat backfill as an emergency script at 2am get to enjoy it as exactly that.

What "designed for backfill" means:

Idempotent processing. Reprocessing a document yields the same result and replaces cleanly. If your pipeline appends, you don't have a pipeline, you have an accumulation.
Versioned indexes with a swap. Backfill into kb_v7 while kb_v6 serves traffic; run your retrieval evals against v7; flip the alias; keep v6 around until v7 has been boring for a week. Rebuilding an index in place under live traffic is how agents spend an afternoon answering from a half-built brain.
Separate lanes for backfill and live traffic. A backfill will happily eat every rate-limit token and embedding-API dollar you have. Throttle it, run it in its own queue, and make sure the 15-minute SLA on live updates still holds while five years of history grinds through.
A backfill eval gate. The reason you're backfilling is usually to make retrieval better — prove it. Golden retrieval queries run against the new index before the swap, same discipline as any other agent change.

The takeaway

Feeding an agent is a pipeline discipline: inventory the sources, sync what's referenced and live-fetch what's stateful, mix events with reconciliation sweeps, scrub PII before it ever enters the system, put numbers on freshness and alert when you miss them, and build the backfill path before you need it. None of it demos well. All of it is why one agent answers from reality and another answers from three weeks ago — and users can tell the difference in about four questions.

Related reading