Measuring RAG Impact Before and After: Four Metrics That Matter

A manufacturing company's quality assurance team used to spend 22 minutes tracking down a single process exception record. The process required pulling information from three different systems, paging two senior engineers, and waiting for responses. They implemented a RAG-based knowledge system. On the first day, everyone agreed it was significantly faster. When finance asked how much time was actually saved and what that translated to in dollars, the team couldn't provide an answer. "Significantly faster" wasn't useful without numbers.

This scenario comes up constantly. RAG's ROI is difficult to articulate not because the system doesn't work, but because most teams never established baseline metrics before launch. By the time you need to defend the investment, there's no "before" snapshot to compare against.

You need to track four metrics, not impressions

A defensible RAG impact framework, one your finance team will accept and auditors can verify, comes down to four metrics. Each one needs paired data from before and after implementation:

Retrieval precision and recall: Given a set of test questions, what percentage of documents your system retrieves are actually relevant (precision), and what percentage of relevant documents were actually found (recall)? This measures the retrieval layer alone, separate from LLM generation. It's the foundation your RAG quality rests on.

Average time to find information: For the same batch of tasks, how many minutes does manual search take versus RAG-assisted search? This is the easiest metric for finance to understand and the easiest to convert into salary dollars.

Answer accuracy: Of the final answers the model produces, what percentage are correct when checked against human review or reference answers? Keep this separate from retrieval accuracy, a missed retrieval and a hallucinated answer are two different failure modes.

Cost per query: The total cost of one complete question-answer cycle, token consumption, vector database lookups, infrastructure costs amortized across queries. This metric determines whether scaling RAG expands your ROI or gets crushed by operating costs.

Before and after data is essential

A client brought us in three months post-launch to measure results. The system looked solid, 87% answer accuracy. Impressive number. But nobody had measured the baseline. We ran the numbers on manual search, and senior staff achieved 91% accuracy. So 87%, was that progress or regression? This question took one day to answer before launch. Measuring it afterward is nearly impossible, and the comparison is unconvincing either way.

The first step is always the same: Before launch, lock in a golden test set of 50 to 100 questions. Make sure it spans three difficulty tiers, high-frequency searches, long-tail queries, and cross-document lookups. Measure your human baseline against this set. Then run this identical test suite every time you iterate. Now you have a real, comparable timeline.

Ready-to-use benchmark template

Copy the table below. Fill in the pre-launch column, then fill in the post-launch column, and your ROI emerges automatically. These numbers come from an actual mid-market financial services customer support team. Substitute your own data:

Metric	Before Launch (Human Baseline)	After Launch (RAG)	Change	Measurement Method
Retrieval precision	Not applicable	82%	Not applicable	Relevance-coded by human review on 100 gold test questions
Retrieval recall	Not applicable	78%	Not applicable	Same method, measured as coverage of documents that should have been found
Answer accuracy	91%	88%	-3pp	Compared against reference answers, dual-reviewed
Avg. time per lookup	22 min	4 min	-82%	Timed on 30 real-world tasks, averaged
Cost per query	Not applicable	$0.06	Not applicable	Token costs + vector search + amortized infrastructure
Monthly query volume	3,000	3,000	Not applicable	System logs
Monthly hours saved	Not applicable	900	Not applicable	(22-4)×3,000÷60
Labor cost savings	Not applicable	$13,500	Not applicable	900 hours × $15/hour
Monthly system cost	Not applicable	$180	Not applicable	$0.06×3,000 + platform fees

The value isn't whether the numbers look impressive. The value is that this table forces you to confront trade-offs. Notice the example shows accuracy actually dropped three percentage points, senior staff judgment still outperforms the model. But search time fell 82%, and new staff now reach near-senior-level performance. That's the real ROI conversation: you're not buying an all-around-better system. You're making a calculated swap, accepting some accuracy loss to gain massive improvements in scale and speed.

How these numbers get inflated

Don't report retrieval accuracy as a single average. Break it down by query type. High-frequency questions score high no matter what, they're not the test. Long-tail questions are where the real system quality shows up. And long-tail is exactly where knowledge systems are supposed to create value.

Don't calculate cost-per-query as just token spend. Amortize the full cost. Demo-day numbers look great until you factor in the vector database, reranker model, monitoring infrastructure, the fixed costs. When query volume is still small, per-query costs can shock you.

Your reference answers have to be locked down and reviewed by two independent people. If the system builders score their own answers, the numbers will look terrific. They'll also mean nothing for decision-making.

The deployment reality

When we do field deployments, we usually help clients build this baseline table before the contract gets signed. It accomplishes two things: post-launch ROI backed by evidence, and clear criteria for whether iteration rounds warrant continued investment. Day-one enthusiasm doesn't matter. What matters is whether these numbers hold up three months later.