Background

The “Topics and Intents Classifier” labels each conversation between an end user and a customer's Ada-powered AI agent, surfacing in the dashboard what people contact the customer about. The legacy classifier generated topics automatically based on a one-sentence inquiry summary, and assigned just one topic to each conversation.

The new system reverses all three: the customer authors the taxonomy (the set of topics and intents that conversations are sorted into), the classifier reads the full transcript, and a single conversation can hold multiple (topic, intent) pairs. That last change matters: one customer measuring containment—how often the agent resolves a conversation without a human handoff—wanted to count conversations where end users requested a human midway through, which the legacy single-topic classifier overlooked, capturing only the initial request. The classifier has been validated on taxonomies with up to several hundred intents.

Experiments & findings

Customer-authored taxonomies with explicit boundary definitions. Classifying into hundreds of intents is a regime where model-side improvements alone tend to plateau. The lift comes instead from how precisely each one is defined, which only the customer truly knows.

The classifier maps each conversation to one or more (topic, intent) pairs, where the topic is the subject area (for example, Billing) and the intent is what the end user wants to accomplish (for example, dispute a charge). Rather than generating these pairs automatically, the customer authors them: each pair gets a written scope, inclusion patterns, and exclusion conditions, expressed in the same plain language a support team already uses in its onboarding documentation. Customers don't start from a blank page: a pipeline drafts an initial taxonomy from their legacy topics and sampled conversations, which they review before activation.

Full-transcript classification for multi-label quality. How much of the conversation the classifier reads determines how well it can assign multiple labels. The legacy system, classified based on a one-sentence inquiry summary, was originally a cost-saving choice from an era of much pricier models. By construction, that summary retains the dominant intent and drops the rest.

If an end user asks why they were charged twice and then asks to cancel their subscription, that summary captures only the billing dispute, and the cancellation intent disappears before classification even runs. Reading the full transcript instead lets the classifier recover the sub-intents that emerge across turns, so it can label all (topic, intent) pairs a conversation may contain rather than only the one it opened with. Capturing all pairs is what lets a customer see patterns the single-topic view could never surface, such as how often a billing dispute ends in a cancellation.

Moving beyond traditional clustering metrics to an LLM-as-judge. Evaluating quality across hundreds of intents is hard. Grouping conversations by their (topic, intent) is effectively clustering those conversations by shared end-user goal, and clustering quality is notoriously hard to measure. Standard clustering metrics like the silhouette score rarely align with human judgment and manual review does not scale.

We reframed evaluation as a classification task an LLM-as-judge can do reliably. Given the conversation, the assigned (topic, intent) pair, and the full taxonomy, the judge determines whether each is defensible under its definition and whether a better pair exists. Topic accuracy (the share of assigned topics the judge upholds) climbed above 90% and intent accuracy 80%, even across taxonomies with more than 150 intents. Because the judge scores only the labels actually assigned, an intent with no good home is flagged as out-of-taxonomy rather than forced into an ill-fitting pair, much as a clustering method like HDBSCAN can leave a conversation unassigned rather than forcing it into the wrong cluster.

Self-improving cluster quality with an agentic loop

Refining a taxonomy of this size by hand is slow, but the judge's output eliminates the need. Because it emits structured, per-case feedback—including when it withholds approval and the better-fitting pair it would have chosen—a coding agent can drive taxonomy and prompt refinement end-to-end rather than waiting on human review cycles. The agent reads the judge's verdicts, proposes a prompt or taxonomy change, re-runs classification, and re-measures against the judge—all without a human in the loop.

In one case the judge surfaced conversations requesting a two-factor reset that had been tagged with the generic "Sign-In Issue" intent when a more specific "Reset Two-Factor Authentication" intent already existed in the taxonomy. The agent merged the two overlapping intents, re-ran the pipeline, and the judge no longer flagged the confusion. Beyond merging intents in the taxonomy, the agent also tightened the classifier prompt for this same kind of confusion, adding a rule to prefer the more specific intent whenever a broader one would also match. Once the loop converges, the refined taxonomy goes back to the customer for review and approval before it goes live; the taxonomies coming out of this loop have been well received by early-access customers.

Market implication

The new system treats conversation classification as an explicit collaboration with the customer rather than something the model decides on alone.

Three patterns from this work generalize beyond Ada:

• First, for multi-label conversational data, classifying from the full transcript rather than a short summary is what makes capturing every intent possible. A summary silently drops the secondary ones.
• Second, reframing cluster quality as a classification question (does this item belong in the cluster it was assigned to?) lets an LLM-as-judge scale quality assessment beyond traditional clustering metrics or manual review.
• Third, when that judge emits structured per-case feedback, a coding agent can drive the entire iteration loop end to end: read the failures, propose prompt or taxonomy changes, re-run, and re-measure, far faster than human review cycles.

The biggest gains in classifying support conversations came less from the model than from the system around it: letting customers author the taxonomy, classifying from the full transcript rather than a one-line summary, and refining the result through an agentic loop that iterates against an LLM-as-judge. Together, these took our early-access customers above 90% topic accuracy and 80% intent accuracy, even on taxonomies with more than 150 intents.