Designing a RAG evaluation framework: golden test sets and quality metrics

Here's a mistake we made.

We had a manufacturing client with an internal knowledge Q&A system for engineering specs, part numbers, and SOPs indexed into a vector database. On demo day: ten questions, ten correct answers. Three weeks after launch, a production line supervisor posted a screenshot: the system reported torque specs for Product Line A using Product Line B's specs. Investigation showed those ten demo questions were hand-selected by engineers. Real users asked in countless ways, and there was no mechanism to measure how many answers were actually wrong.

This is why the evaluation framework comes before optimization. Without measurement, every prompt change, embedding swap, and chunk size adjustment is just guessing on feel.

RAG evaluation framework: define what you're measuring

An evaluation framework divides 'how well an answer works' into independently observable dimensions. RAG failures come from retrieval or generation. Mixing them obscures whether to fix the knowledge base or the model. Step one: separate them.

On the retrieval side, track two things. Recall: do the passages that should be found make it into the top-k results? This is where hallucinations begin, when data isn't retrieved, the model fills gaps. Context precision: of the passages retrieved, how many are actually relevant? Too much noise pulls the model off track.

On the generation side, also two. Faithfulness: can every claim in the answer find support in the retrieved context? This is the core defense against hallucination. Answer relevance: does the answer address the user's question, or does it just recite something technically correct but beside the point?

These four dimensions form the minimal evaluation scaffold.

Golden test sets: the foundation of everything

Without a golden test set, all metrics above remain abstract. A golden set is human-verified triples, question, reference answer, source passages it should retrieve from, serving as ground truth for regression tests.

Building one follows three principles. First, questions come from real users, not engineered examples. Without production traffic, review support tickets, internal Q&A logs, Slack threads. Second, include hard cases: multi-hop reasoning, cross-document synthesis, questions where the correct answer is 'not in the data' (these surface hallucinations most effectively). Third, optimize for depth over breadth. Fifty to a hundred well-annotated questions beat a thousand vague ones. Always record which passages should be retrieved, otherwise recall cannot be calculated.

Keep the golden set in version control alongside code. Review it in PRs. Every time the knowledge base updates or the model changes, run it again and check whether metrics regressed. That is what regression testing means.

LLM-as-judge: scaling evaluation

Semantic judgments like faithfulness and relevance cost too much to hand-label at scale. The practical approach is LLM-as-judge: use a capable model as arbiter, scoring each answer against a rubric.

Building trustworthy judges requires precise prompt engineering: give explicit scoring criteria, require sentence-by-sentence comparison against context, output structured scores and reasoning instead of 'seems okay.' Binary or ternary judgments (does this have support?) prove far more stable than graded scores (rate 1-5). Design for binary when possible. Validate the judge by spot-checking 20% against human review, measuring agreement rate, and rewriting the rubric if it falls short.

Which judge model you select directly affects cost and stability. Run a small-scale comparison before committing.

A metrics template

These thresholds are not fixed rules, calibrate them to your context. Healthcare and finance should push faithfulness close to 1.0; internal Q&A can be looser. Set the numbers first, then measure every change against them.

Make evaluation part of your operations

Evaluation is not a pre-launch checkpoint, it is continuous. Plug the golden set into CI so every prompt change, model swap, and knowledge base update automatically triggers the full metric suite. Block any deployment where a dimension regresses. In parallel, continuously sample real conversations in production, feed new hard cases back into the golden set, and let it grow with the business.

The manufacturing customer mentioned earlier showed faithfulness at 0.78 on first regression. The issue: chunks cut too small, splitting part numbers and torque specs across boundaries. Adjusting the chunking strategy brought faithfulness to 0.96, and error reports ceased.

Golden test sets and eval processes are first-class deliverables that ship with the engineering team, not documents added afterward. A polished demo proves nothing. What matters is measurement, regression testing, and daily production use.