Corrective RAG for Billing Questions on WhatsApp

Most RAG demos are graded by an audience that cannot check the answer. Ask a documentation bot something and a fluent, plausible paragraph comes back — nobody in the room knows whether the third sentence is invented, so the demo lands.

Billing is not like that. When a subscriber asks “why is my bill higher this month,” they are holding the invoice. They will check. A confidently wrong answer about someone’s money is not a soft failure that shows up in an eval later; it is a support ticket, a chargeback dispute, and a customer who now trusts the channel less than before you automated it.

We built a WhatsApp assistant for ABDO 77, an ISP in Guigue, Carabobo, that answers plan and billing questions. It has two sources: a knowledge base of markdown documents, retrieved with corrective RAG, and an API call that pulls the specific subscriber’s data when the answer depends on their account.

That split is the whole design, and it is the thing most implementations get wrong — not because the split is hard to build, but because the two sources fail in completely different ways and people apply one mitigation to both.

The failure mode naive RAG cannot see

Standard retrieval-augmented generation is a pipeline with no self-doubt: embed the question, pull the top-k chunks, stuff them into the prompt, generate. The model is instructed to answer from context, and it will — whether or not the context contains the answer.

That is the whole problem. The pipeline has no step where anything asks “was that retrieval any good?” A query that matches nothing useful still returns k chunks, because top-k always returns k. The model then does what models do with thin context: it produces something fluent from adjacent material.

For a docs bot that degrades into vagueness. For billing it degrades into a specific, checkable, wrong number.

Plan and invoice questions are also unusually hostile to embedding similarity. “Why did my bill go up” is semantically close to the tariff page, the promotions page, the terms of service, and the prorating policy — all four are about charges, and only one explains this subscriber’s delta. Retrieval will happily hand you the confident-sounding wrong neighbour.

What corrective RAG actually adds

Corrective RAG (Yan et al., 2024) inserts a step that naive RAG lacks: a lightweight retrieval evaluator that scores the retrieved documents against the query before generation, and emits a confidence verdict — the paper uses Correct, Incorrect, and Ambiguous. What happens next branches on that verdict rather than always running the generator.

The mechanism that matters in production is not the accuracy bump on benchmarks. It is that the architecture has somewhere to put the “I don’t know.” Naive RAG structurally cannot decline: there is no branch for it. CRAG creates the branch, and then it is your job to decide what lives there.

Our reason for reaching for it was general: corrective retrieval is the current technique for letting a RAG pipeline catch its own bad retrievals instead of generating over them. What made it non-optional rather than nice-to-have was the domain. In most applications a self-correcting retriever improves an average. In billing it is the difference between “I’ll get someone to check that” and a wrong number about money.

The paper’s own answer to a bad verdict is to broaden retrieval — web search as an extension when the local corpus comes back weak, plus a decompose-then-recompose step to strip irrelevant material out of the chunks that did survive. For a billing assistant, broadening to the open web is exactly wrong: the answer to “why is my bill higher” is never on the internet, it is in this operator’s rate card and this subscriber’s account. So here the Incorrect branch does not reach further. It stops and hands off.

There is now also an independent open-source reproduction of CRAG with an explainability analysis, which is worth reading before you commit — the retrieval evaluator is the component your whole refusal policy hangs on, and knowing where it is confidently wrong is more useful than knowing its average score.

Two sources, two failure modes, two mitigations

Here is the mistake worth saving you: treating an API lookup as if it were another retrieval.

They are not the same kind of thing, and the difference is not academic.

The markdown corpus is retrieved by similarity. It can hand back the confidently wrong neighbour — the promotions page when you asked about prorating. The failure is relevance, and relevance is exactly what a retrieval evaluator is built to grade. This is CRAG’s job and it does it well.

The subscriber API is a keyed lookup. You pass an identifier and get that account’s data back. Grading it for relevance is meaningless: it is not “probably about” the right subscriber, it either returned the right record or it errored. There is no Ambiguous verdict to compute, and running a relevance evaluator over it buys you nothing.

But the API introduces a failure mode that retrieval does not have, and it is easy to miss because the data going in is correct: the model can narrate accurate numbers inaccurately. Hand an LLM a current balance, a previous balance and three line items and ask it to explain the difference in Spanish, and you have asked it to do arithmetic and attribution in prose. It will sound completely certain either way. The retrieval evaluator cannot help you — nothing was retrieved badly.

So the mitigations do not transfer:

	Markdown corpus	Subscriber API
How it’s reached	Similarity search	Keyed lookup
How it fails	Retrieves the wrong-but-adjacent document	Returns correct data the model then describes wrong
What catches it	Retrieval evaluator, refusal branch	Never let the model compute the figures — pass through what the API returned
What doesn’t help	—	A relevance grade

The general principle: numbers that came from a system of record should reach the subscriber as those numbers, not as the model’s summary of them. The model’s job is the sentence around the figure, not the figure. Anything requiring a delta, a total or a date comparison is a computation, and a computation belongs in code where it is testable — not in a paragraph where it is merely fluent.

This is the part we would push hardest on in a review of anyone else’s billing assistant, because it survives every model upgrade. A better model narrates wrong arithmetic more persuasively.

WhatsApp removes the affordances you would normally use

The standard mitigation for retrieval uncertainty is to show your work: inline citations, a sources panel, a hover card with the passage, a confidence badge. Web chat interfaces are full of places to put hedging.

WhatsApp has none of them. You get plain text, in a thread the user reads on a phone, in a channel they associate with talking to people. There is no footnote. There is no sidebar. A long URL pasted mid-sentence reads as spam, not as a citation.

Two consequences follow, and both are design constraints rather than implementation details:

Hedging has to be structural, not typographic. You cannot dim a low-confidence sentence or attach a superscript. The only honest signal available is what the message says and whether it says a number at all. Which means the confidence decision has to happen before generation — exactly what the evaluator gives you — because you cannot retrofit doubt into a sentence that already asserts a figure.

The escape hatch has to be a person. In a web app, “I’m not sure” can degrade to a search box or a docs link. On WhatsApp, the user is already in a conversation; the natural degradation is handing that conversation to a human agent with the context attached. This is the part that makes the whole thing an operations project and not just an AI project — the refusal path needs somewhere to land, and that means the support rota is part of your architecture.

The refusal path is the product

The instinct is to treat refusals as the failure rate and drive them to zero. That optimises the wrong thing. Every refusal you eliminate by loosening the evaluator becomes a confident answer about a bill, and a subset of those are wrong in a way the customer will notice.

The framing that survives contact with an ops team: the assistant’s job is not to answer everything, it is to absorb the volume that is genuinely answerable and hand off the rest cleanly. Plan comparisons, what a line item means, when the cycle closes, what a promotion covered and when it lapsed — that is real, repetitive volume with verifiable answers sitting in documents. It is worth automating.

“Why is my bill $14 higher than last month” is a different question. It needs the subscriber’s account, not the rate card — answer it from documents and you are wrong for every subscriber whose delta came from something else. That is what the API call is for.

Knowing which of the two you are looking at is the actual engineering problem, and it happens before either source is touched. It is a routing decision, and the retrieval evaluator is what makes it possible to route at all.

In practice the handoff fires on three conditions, and only the first is the clever one:

1. The evaluator returns a bad verdict on what it retrieved.
2. The question isn’t in the knowledge base at all — no document covers it, so there is nothing to be confident about.
3. The user asks for a person. Unconditionally, immediately, no attempt to satisfy them first.

That third one is worth stating plainly because it is the one teams argue about. There is always pressure to make the bot try once more before releasing the conversation — it improves your containment metric. It also means the subscriber who typed “I want to talk to a human” gets a paragraph they did not ask for. On a channel people use to talk to people, ignoring that request is how you teach them the bot is an obstacle rather than a front door.

What to measure

Accuracy on a static eval set is necessary and nowhere near sufficient, because the interesting failures are the ones where retrieval was confidently wrong. Four things worth instrumenting from day one:

• Refusal rate, tracked deliberately — as a dial you tune against complaints, not a defect to minimise
• Handoff quality — when a human takes over, did they get the context, or do they start from zero and re-ask everything
• Verified-wrong answers — the count that actually matters, and the only one your customer feels; it comes from support, not from your eval harness
• Evaluator disagreement — sample where the retrieval verdict and the eventual human resolution diverge; that set is where your next fix lives

The general shape of this — evals, cost controls, guardrails — is the same discipline any production AI system needs. Billing just removes your margin for error, which makes it a good forcing function.

Where this applies

Any operator with high-volume, repetitive, document-answerable questions and a channel where users expect a human: ISPs, utilities, insurers, clinics, property managers. The common shape is a documented rate structure, a subscriber base that asks the same forty questions, and a support team drowning in the answerable ones.

If that describes your operation, tell us about the question volume and we will tell you which slice is worth automating and which slice needs account state — the second one is usually where projects like this go wrong.

One honest note on scope: this is one system for one operator, described as architecture rather than as a benchmark. We are not going to show you a deflection-rate chart from a sample of one. What we can show you is the reasoning, and three other systems we built that are still running.