RAG Evaluation Frameworks: Reference Data, Retrieval Metrics and Judge Calibration

Define the task and reference scope

Specify question types, approved sources, user permissions and output. Record source, model, prompt and processing versions. Retrieval and generation are useful diagnostic layers alongside source errors, applicability, freshness and pipeline failures.

Faithfulness measures support by supplied context. A wrong, outdated or inapplicable source can yield a faithful but wrong answer. Microsoft's RAG evaluation guide distinguishes grounding from correctness; assess both.

Version reference data

Record reference answers, acceptable equivalents, expected passages and applicable conditions. Include cross-document, insufficient, conflicting, denied-source and pending cases. Review labels and disagreements rather than treating references as immutable truth.

Use approved, necessary questions/records with task groups and source versions. Owners review applicability after changes. Separate development cases from held-out release evidence. Choose coverage for variation, risk and the question being tested, not a universal sample count.

Specify metric implementations

The precision/recall below use unranked passage counts, not every library's identically named metric.

Declare per-query/aggregate methods, weights and exclusions. Empty denominators or claim-free answers must not silently become perfect scores. Correct abstention on insufficient-evidence cases differs from overall request refusal.

Calibrate model judges against human references

If using LLM-as-judge, record model, prompt, rubric, reference and source versions. Define supported, unsupported and unresolved labels with structured results and checkable reasons.

Compare representative human judgments for agreement and errors, including consequential failures, task groups and disagreements. Evaluate binary, graded or other formats for the use. A fixed review share or confident rationale does not establish reliability. Recheck comparability after model/rubric changes.

Use regression to support release

Set criteria from task consequences, baseline, metric implementation, groups and uncertainty. Internal specifications can be consequential; sector labels alone do not justify looser quality. High faithfulness is not professional authority or external truth.

CI can compare candidate regressions, but small single-run changes may reflect model/judge variability. Define block, investigate, repeat or review conditions instead of universal numeric gates. Check purpose, labels and versions before adding operating cases; tuned examples are not independent release evidence.

A PoC provides partial evidence. Operation also requires source-update, failure and review ownership. NIST AI RMF (January 26, 2023) offers voluntary contextual monitoring. Anthropic's hallucination guidance offers grounding checks without eliminating errors.

Sources reviewed October 4, 2026. Microsoft's guide was updated November 21, 2025; Anthropic displays no update date. See the RAG engineering stages or define evaluation scope through the RAG service.