8 metrics for RAG answer quality: acceptable thresholds and calculation formulas

RAG answer quality uses quantifiable, auditable metrics to assess how trustworthy a system's responses are. Specifically: does each claim come from something the system actually retrieved? Do citations point to supporting evidence? Does the system know when to refuse an answer?

Why use metrics instead of a subjective feeling? Because 'answers well' is a feeling, and you can't monitor feelings once the system launches.

A financial services company had an internal Q&A system. For the pre-launch demo, the manager hand-selected 20 questions and the system answered all of them. When we ran it against 300 real customer service questions from their actual history, faithfulness dropped to 0.82. That means roughly 1 in 5 sentences was something the model had made up, and made it convincingly. The model wasn't the problem. Nobody had defined what 'acceptable' meant for this system.

8 metrics for RAG answer quality and acceptable thresholds

These 8 metrics go into customer acceptance agreements and run daily after systems launch. The thresholds below work for enterprise knowledge scenarios: compliance, healthcare, manufacturing SOPs. High-risk domains should set them higher.

How to evaluate whether a claim is supported

The first four metrics hinge on breaking answers into atomic claims: individual statements that can independently be verified as true or false. A typical answer contains several claims. For each, ask: Can this claim be found in the context chunks the retrieval step returned? If yes, the claim is supported. If no, it's a hallucination.

You can use a stronger model as a judge (LLM-as-judge), but don't trust it completely. One approach: pull 10% of results for human review and use that to calibrate the judge model's consistency. If judge and human annotator agree less than 90% of the time, those scores aren't reliable.

Citation precision and attribution coverage confuse teams easily. High attribution coverage with low citation precision means every claim has a source attached, but the source doesn't actually support it. That's worse than having no citation at all, because it creates an illusion of credibility.

Refusal as a feature, not a failure

Correct abstention is the most overlooked metric. In enterprise contexts, 'I don't know' often matters more than a fabricated answer. Track two numbers side by side: how often the system correctly refuses genuinely unanswerable questions (should be high), and the false refusal rate, questions it should have answered but didn't (aim for ≤ 0.05). If you only track the first, the system learns to refuse everything to improve the score.

Don't pursue a single composite score. Customer acceptance dashboards track these 8 metrics separately, each on its own. Faithfulness and hallucination rate are red lines: fail either and deployment is blocked. Low context recall? Fix retrieval and chunking first, not the model. When metrics point to levers you can actually move, the team knows what to fix.

One principle applies in every RAG implementation: define your acceptance criteria before you write the first line of code. After launch, these 8 metrics run automatically every day with someone watching. A perfect demo means nothing. What matters is whether the system still meets the bar three months later and whether anyone is actually using it.