Webhooks for Event-Driven AI Agents: The Six Problems You'll Hit

The most useful agents I've deployed don't wait to be asked. A support ticket lands, a payment fails, a call recording finishes processing — and the agent reacts. The plumbing behind that is almost always a webhook: the external system POSTs you an event, and your agent pipeline picks it up.

Webhooks look trivial. An HTTP endpoint, a JSON body, done. But there's a standard set of six problems that every webhook consumer eventually hits, and when the consumer is an agent — something that takes actions based on the event — each problem gets sharper. An unverified webhook isn't just bad data; it's an instruction injected into a system that can act. I've built these pipelines across voice-agent platforms and automation deployments, and this is the walkthrough I give engineers before they wire the first one.

Let's follow one event through the gauntlet.

Problem 1: Is this event even real?

Anyone who discovers your webhook URL can POST to it. If that payload flows into an agent's context, you've built a prompt-injection front door with a public address.

Verify signatures on every delivery, before you parse anything. Most providers sign the raw body with a shared secret — HMAC-SHA256 is the standard shape:

import { createHmac, timingSafeEqual } from "node:crypto";

function verifySignature(rawBody: Buffer, header: string, secret: string) {
  const expected = createHmac("sha256", secret).update(rawBody).digest("hex");
  const a = Buffer.from(expected);
  const b = Buffer.from(header);
  return a.length === b.length && timingSafeEqual(a, b);
}

Three details that bite people: sign the raw body (any framework that parses JSON first will re-serialize it differently and break verification), use a constant-time comparison, and check the timestamp most providers include to block replay attacks. And even after verification, treat the payload as untrusted content when it reaches the model — a verified webhook from a ticketing system can still contain a customer-written message that says "ignore your instructions."

Problem 2: Answer fast, work later

Providers give you seconds — often fewer than five — to return a 2xx before they mark the delivery failed. An agent invocation takes longer than that on a good day. So the endpoint itself must do almost nothing:

Verify the signature.
Enqueue the event (SQS, a Postgres outbox table, whatever you already run).
Return 200.

All agent work happens in a worker consuming the queue. This one structural decision quietly solves half the remaining problems: you get buffering during traffic spikes, natural retry semantics on your side, and the provider stops timing you out because your LLM call took eleven seconds.

Problem 3: You will receive this event twice

Webhook delivery is at-least-once, everywhere, always. Providers retry on timeouts, networks duplicate, and your own queue redelivers. For a passive consumer, duplicates are noise. For an agent, a duplicate means the customer gets two follow-up emails or the CRM gets two identical call logs.

Idempotency keys are non-negotiable. Use the provider's event ID (or a hash of the payload if there isn't one) and record it before processing:

INSERT INTO processed_events (event_id, processed_at)
VALUES ($1, now())
ON CONFLICT (event_id) DO NOTHING;
-- zero rows inserted → duplicate → drop it

And push the same discipline downstream: the agent's writes should carry idempotency keys too, so that even if processing does run twice, the side effects land once. Dedupe at the entrance, idempotency at the exits.

Problem 4: Retries, and where events go to die

Your worker will fail — a downstream API is down, a payload is malformed, the model returns something unparseable. The playbook:

Retry with exponential backoff for transient failures. Three to five attempts covers most upstream blips.
A dead-letter queue for everything else. After max retries, the event goes to a DLQ with the error attached — not into a log line that scrolls away.
Alert on DLQ depth, and build redrive from day one. A DLQ nobody watches is just a slower way of dropping events. You want to fix the bug, then replay the dead events through the normal path.

One distinction worth encoding explicitly: retryable versus permanent failures. Retrying a malformed payload five times is theater. Classify the error at the point you catch it, and route permanent failures straight to the DLQ with a reason.

Problem 5: Ordering is a lie

Nobody guarantees webhook order. ticket.updated can arrive before ticket.created. Two updates to the same record can arrive swapped. If your agent builds its picture of the world by applying events in arrival order, that picture will eventually be wrong — and the agent will act on it confidently.

Two patterns that hold up:

Treat the event as a doorbell, not the data. On receipt, fetch the current state of the object from the API and act on that. You trade an extra API call for immunity to ordering, staleness, and skinny payloads. For agent use cases this is my default — the agent should reason over the truth, not over a possibly stale snapshot.
Version checks where fetching is expensive: providers often include an updated_at or sequence number; ignore any event older than the state you've already processed.

If you genuinely need per-entity ordering (rare), partition your queue by entity ID — FIFO queues with message groups — and accept the throughput ceiling that comes with it.

Problem 6: Sometimes the answer is polling

Contrarian but true: webhooks aren't always the right tool.

The system has no webhooks. Plenty of enterprise and legacy software doesn't. Poll on a schedule and diff against last-seen state — a boring cron beats a heroic workaround. (More on that world in integrating agents with legacy systems.)
You need completeness guarantees. Webhooks get dropped — provider incidents, your outage window, an expired secret. Any pipeline where missing an event has real cost needs a reconciliation loop anyway: a periodic poll that compares provider state to yours and heals gaps. Once that loop exists, the webhook is just a latency optimization on top of it.
Low volume, weak latency needs. If a five-minute delay is fine and you get a dozen events a day, polling is less infrastructure and fewer failure modes.

My rule of thumb: webhooks for speed, polling for truth, both when the stakes are high. In one deployment we ran the reconciliation poll nightly and assumed it was dead weight — until a provider silently disabled our webhook subscription after a cert rotation. The poll caught it the next morning; without it, the agent would have gone quietly deaf for weeks while everyone assumed things were just slow.

The shape of a pipeline I trust

Verified endpoint → queue → idempotent worker → agent with narrow tools → idempotent writes, with a DLQ and a reconciliation poll bolted on. Every piece is boring. That's the point — the interesting behavior should live in the agent, not in the plumbing surprising you.

Event-driven agents are worth the ceremony. They feel proactive to users, they cut the polling waste, and they let one integration serve many workflows. Just respect the six problems — because the failure mode of each one is an agent acting on bad input, and that's a much worse Tuesday than a dropped JSON blob.

Related reading