Corrective RAG for Billing Questions on WhatsApp

Most RAG demos are judged by people who can’t check the answer. A fluent paragraph comes back, and nobody in the room knows whether the third sentence is made up.

Billing doesn’t give you that cover. When a subscriber asks “why is my bill higher this month?”, the invoice is open on the same phone. They will compare. A confident wrong answer about someone’s money turns into a support ticket, a dispute, and a customer who trusts the WhatsApp line less than before you automated it.

We built a WhatsApp assistant for ABDO 77, an internet provider in Güigüe, Carabobo, that answers plan and billing questions. Corrective RAG (CRAG) is retrieval-augmented generation with a checkpoint: before the model writes anything, an evaluator grades whether the retrieved documents answer the question, and a bad grade leads to a different action instead of an answer.

The assistant has two sources of truth, and they need different defenses. The markdown knowledge base (plans, rates, policies) can return the wrong page, so it goes through corrective RAG with a branch that hands off to a person. The account API returns the right data, but the model can describe it wrong, so every figure is computed in code and the model only writes the sentence around it.

Why plain RAG gets billing wrong

The textbook pipeline never doubts itself: embed the question, pull the top-k chunks, put them in the prompt, generate. Top-k always returns k results, even when none of them answer the question. The model is told to answer from context and it does, stitching something plausible out of whatever is nearby. On a docs bot that produces vague answers. On billing it produces a specific number the user can prove wrong.

Billing questions are also hard for embedding similarity. “Why did my bill go up” sits close to the rate card, the promotions page, the terms of service and the prorating policy. All four talk about charges. Only one explains this subscriber’s increase, and often none does, because the reason lives in the account.

Some ordinary retrieval hygiene comes first:

• Chunk by the document’s own sections. Markdown already has headings. One whole section (“Prorated charges when you change plans”) beats a fixed token window cut mid-rule.
• Keep metadata on every chunk (source file, section, last updated) so you know which chunks go stale when a rate changes.
• Use hybrid search. Plan names, codes and amounts are exact strings, and vector search is weak at exact matches. Combine it with keyword search (BM25), merge the rankings with reciprocal rank fusion, then rerank.

None of that lets the pipeline say “I don’t know”. That’s what corrective RAG adds.

What is corrective RAG (CRAG)?

Corrective Retrieval Augmented Generation (CRAG), by Shi-Qi Yan, Jia-Chen Gu, Yun Zhu and Zhen-Hua Ling (2024), puts a retrieval evaluator between retrieval and generation. In the paper it’s a fine-tuned T5-large, about 0.77B parameters, that scores each retrieved document against the question. Two thresholds turn those scores into one of three actions:

• Correct: at least one document scores above the upper threshold. The documents are refined with a decompose-then-recompose step that keeps the relevant strips and drops the rest.
• Incorrect: every document scores below the lower threshold. The paper discards the retrieval and goes to web search.
• Ambiguous: anything in between. The paper combines both kinds of knowledge.

For a production system the useful part is structural: the pipeline now has a branch where “I don’t know” can live, and what goes there is a product decision.

For billing, the paper’s answer to a bad verdict is the wrong one. The reason a bill went up is never on the open web. It’s in this operator’s rate card and this subscriber’s account. So in our case the Incorrect branch doesn’t search further. It stops and hands the conversation to a person.

Before you trust the evaluator, read the independent open-source reproduction (Yalavarthi, 2026). Its explainability analysis finds that the T5 evaluator leans mostly on named-entity alignment rather than semantic similarity, and that it transfers poorly outside its training domain. Billing messages are short on named entities (“why is it higher?”), so build your evaluation set from real subscriber messages, not from tidy questions that mention the plan by name.

An API lookup is not retrieval

Treating the account API as one more retriever is the mistake I’d most like to save you from.

The markdown corpus is searched by similarity, so it can return an adjacent, wrong document: the promotions page when the question was about prorating. That’s a relevance failure, and grading relevance is what the evaluator does.

The subscriber API is a keyed lookup. You send an account ID and get that account back, or an error. There’s nothing to grade and no Ambiguous state.

But the API brings its own failure, easy to miss because the data is correct: the model can narrate correct numbers incorrectly. Give an LLM a current balance, a previous balance and three line items, ask it to explain the difference in Spanish, and you’ve asked it to do arithmetic in prose. It sounds equally sure either way, and the evaluator can’t help, because nothing was retrieved badly.

	Markdown corpus	Subscriber API
How you reach it	Similarity search	Keyed lookup
How it fails	Returns an adjacent, wrong document	Returns correct data the model describes wrong
What catches it	Evaluator plus a refusal branch	Compute figures in code; the model only phrases them

The rule I’d apply to any billing assistant: numbers from a system of record reach the customer as those numbers. Differences, totals and date comparisons are computations, and they belong in code you can test. Switching to a stronger model leaves the problem in place, with the wrong arithmetic sounding more convincing.

The routing, sketched

A simplified version of the decision flow in TypeScript. It shows the shape, not the production code; the helpers stand in for whatever your stack uses.

type Verdict = 'correct' | 'ambiguous' | 'incorrect';

async function reply(msg: InboundMessage): Promise<Outbound> {
  // A request for a person always wins. No "let me try once more".
  if (asksForHuman(msg.text)) return handoff(msg, 'user_asked');

  const route = await classifyIntent(msg.text); // 'account' | 'policy' | 'other'
  if (route === 'other') return handoff(msg, 'out_of_scope');

  if (route === 'account') {
    const account = await billingApi.invoiceSummary(msg.subscriberId);
    if (!account) return handoff(msg, 'lookup_failed');
    return explainAccount(account);
  }

  const docs = await hybridSearch(msg.text, { k: 20 }).then(rerank);
  const verdict: Verdict = await gradeRetrieval(msg.text, docs);

  if (verdict === 'incorrect') return handoff(msg, 'low_confidence');
  if (verdict === 'ambiguous') return askToClarify(msg, docs);
  return answerFromDocs(msg.text, refine(docs));
}

And the part that keeps arithmetic out of the model. Amounts are integer cents, the difference is computed here, and the model’s draft is rejected if it contains a number it wasn’t given:

function explainAccount(a: InvoiceSummary): Outbound {
  const deltaCents = a.currentTotalCents - a.previousTotalCents;
  const changed = a.lines.filter((l) => l.amountCents !== l.previousAmountCents);

  const facts = {
    delta: formatMoney(deltaCents),
    lines: changed.map((l) => ({
      label: l.label,
      before: formatMoney(l.previousAmountCents),
      now: formatMoney(l.amountCents),
    })),
  };

  return phraseWithModel('account_delta', facts, {
    // Any number in the draft must come from `facts`; otherwise use the fixed template.
    validate: (draft) => numbersIn(draft).every((n) => allowedNumbers(facts).has(n)),
    fallback: () => renderTemplate('account_delta', facts),
  });
}

The check is crude on purpose: a regex over the draft, a set of allowed values, a fixed template when they don’t match. It catches the failure that matters, a figure nobody gave the model.

What WhatsApp takes away

On the web you show uncertainty with interface: inline citations, a sources panel, a confidence badge. WhatsApp gives you a text bubble in a thread people use to talk to people. There’s no footnote, and a long URL mid-sentence reads like spam. The only honest signal left is what the message says and whether it states a number at all, so the confidence decision has to happen before generation. That’s where the evaluator sits.

The platform does give you two things worth designing around:

• Reply buttons. The Cloud API supports interactive reply buttons: up to three per message, titles of 20 characters or fewer. That suits the Ambiguous branch (“your plan, a charge, or a promotion?” as three taps) and a “Talk to a person” button under every account answer.
• The customer service window. Per Meta’s documentation, a user’s message opens a 24-hour window for free-form replies. After it closes, only pre-approved templates can be sent. A handoff that sits in a queue until tomorrow can miss that window, so the human side needs staffed hours and a template ready for late replies.

The refusal branch needs somewhere to land, which makes the support schedule part of the architecture.

Refusals are part of the product

The instinct is to treat refusals as a failure rate and push them toward zero. Every refusal you remove by loosening the evaluator becomes a confident answer about a bill, and some of those will be wrong in ways the customer notices.

The assistant’s job is to absorb the questions that have an answer in the documents (plan comparisons, what a line item means, when the billing cycle closes, when a promotion ends) and hand off the rest cleanly. In the sketch, handoff fires in four places: the user asks for a person, the question is out of scope, the lookup fails, or the evaluator rejects the retrieval.

Teams argue about the first one. There’s always pressure to let the bot try once more, because it improves the containment metric. It also means someone who typed “I want to talk to a person” gets a paragraph they didn’t ask for, and learns the bot is in the way.

A handoff should carry context: the question, the route, the verdict, the documents or account data used. If the agent has to open with “how can I help you?”, the subscriber explains everything twice.

What to measure from day one

1. A frozen evaluation set of real subscriber questions, each labeled with the expected route. Rerun it on every prompt, model or retrieval change.
2. One trace per conversation with route, verdict, retrieved chunks, latency and cost, so you can see which step failed.
3. Refusal rate as a dial you tune against complaints, not a number to minimize.
4. Answers support marked as wrong. That’s the number the customer feels, and it comes from the support team, not your eval harness.
5. Evaluator disagreements: cases where the verdict and the human resolution diverge. Your next fix is usually in that sample.

This is one system for one operator, described as architecture, so there’s no deflection-rate chart here. The same shape fits any operation with a documented rate structure, customers who ask the same few dozen questions, and a channel where people expect a human: utilities, insurers, clinics, property managers.

If you’re building one, write the frozen evaluation set first, from real messages copied out of your support inbox, before you touch a prompt. If you’d like help sorting which of those questions need account data, write to me.