ASR intent detection with Claude

ASR transcripts are noisy — homophones, half-words, environment noise. A small LLM running as a post-ASR classifier with constrained JSON output cleanly separates "what was said" from "what was meant." This is the exact pattern that took intent accuracy from 41.7% → 91.7% in my paper.

The classifier

import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";

const client = new Anthropic();

const Intent = z.object({
  intent: z.enum([
    "book_appointment",
    "cancel_appointment",
    "ask_pricing",
    "speak_to_human",
    "smalltalk",
    "unclear",
  ]),
  confidence: z.number().min(0).max(1),
  slots: z.record(z.string()).optional(),
  rationale: z.string().max(120),
});

const SYSTEM = `You classify user utterances from an ASR system into one of these intents:
- book_appointment, cancel_appointment, ask_pricing, speak_to_human, smalltalk, unclear

Rules:
- Output JSON only. Match the schema.
- "unclear" if confidence < 0.5 — DO NOT guess.
- Slots: extract date, time, name when present.
- Confidence reflects ASR uncertainty too (correct for obvious mishears).`;

export async function classify(transcript: string) {
  const res = await client.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 256,
    system: [{ type: "text", text: SYSTEM, cache_control: { type: "ephemeral" } }],
    messages: [
      { role: "user", content: `Transcript: "${transcript}"\n\nReturn JSON.` },
    ],
  });

  const text = res.content
    .filter((b): b is Anthropic.TextBlock => b.type === "text")
    .map((b) => b.text)
    .join("");

  return Intent.parse(JSON.parse(text));
}

How it works

The classifier sits after automatic speech recognition. ASR has already turned audio into a (possibly garbled) string; this step decides what the user actually meant. The Zod Intent schema is the contract: an intent drawn from a fixed enum, a confidence between 0 and 1, optional slots, and a short rationale. Defining it once gives both a description of the shape the model should return and a runtime validator for what it actually returned.

The system prompt does the real work. It enumerates the valid intents, states the output rules — JSON only, match the schema, prefer unclear over guessing below a confidence threshold, and account for ASR mishears — and it's wrapped in a cache_control: { type: "ephemeral" } block so the taxonomy and rules are cached across calls and each request only pays for the transcript. The transcript itself goes in the user message. After the call, the code pulls the text blocks out of the response, concatenates them, parses the JSON, and runs it through Intent.parse(...) — so anything that doesn't match the schema throws at the boundary rather than flowing downstream as malformed data. Slot extraction (date, time, name) happens in the same call, which folds what would otherwise be a separate named-entity step into the one request.

Why this works

• Haiku, not Opus. This is a routing call — speed matters more than reasoning depth. Median ~180 ms TTFT.
• Cached system prompt. The full taxonomy + rules sit in cache; per-call cost is just the transcript.
• unclear as a first-class output. Forcing the model to admit uncertainty is what killed the false-positive rate. A hallucinated book_appointment is worse than asking "could you repeat that?"
• Slot extraction in the same call. No need for a separate NER step — Haiku is competent at JSON-schema-conditioned extraction.

Eval harness

const cases: { transcript: string; expected: string }[] = JSON.parse(
  await Deno.readTextFile("./eval/golden.json")
);

let correct = 0;
for (const c of cases) {
  const out = await classify(c.transcript);
  if (out.intent === c.expected) correct++;
}
console.log(`accuracy: ${((correct / cases.length) * 100).toFixed(1)}%`);

Run this every time you touch the system prompt or change the model. Without an eval set, prompt edits are vibes-based.

Notes & gotchas

The single biggest design decision is treating unclear as a first-class outcome. A voice system that confidently mis-routes a garbled utterance to book_appointment is worse than one that asks the caller to repeat themselves, so the prompt explicitly tells the model to fall back to unclear rather than guess. That, more than anything, is what keeps the false-positive rate down.

A few practical things to watch. The code parses the response text as JSON and then validates it with Zod, so you should be ready for two distinct failure modes: text that isn't valid JSON, and JSON that parses but doesn't match the schema (Intent.parse throws on the latter). In production, wrap the parse step and treat any failure as unclear rather than letting it crash the call path. Keep the enum and the system prompt's intent list in sync — if they drift, the model can emit a label the schema rejects. The eval harness is what makes the prompt safe to change: keep a golden set of transcript/expected pairs and re-run it on every prompt or model change, because intent classification is exactly the kind of task where a small wording tweak can quietly move accuracy in either direction.