We've built RAG systems that work.
Not demos. Not proofs-of-concept that fall apart with real users. Production RAG systems that handle thousands of queries daily across legal documents, medical records, technical documentation, and customer support knowledge bases.
The difference between our successful RAG deployments and the ones we had to rebuild? Evals from day 1.
This isn't theory. This is what we learned building 8 RAG systems across Grey Haven portfolio companies, where the pattern is crystal clear:
RAG systems WITH evals:
- Reached production in 6-8 weeks
- Shipped with 85%+ quality scores
- Iterate weekly with confidence
- Catch issues before users do
RAG systems WITHOUT evals:
- Took 16-20 weeks to stabilize
- Deployed with unknown quality
- Play whack-a-mole with bugs
- Learn about failures from angry users
Let me show you exactly how we build evaluation into RAG systems.
The Problem: RAG Fails in Ways You Can't "Feel"
Traditional software has clear failure modes. Either the button works or it doesn't. The API returns data or throws an error.
RAG systems? They're subtly wrong.
Here's what happened at one of our portfolio companies building a legal contract analyzer:
Week 2: Demo looks amazing. CEO shows it to customers. Week 4: Customer reports "AI missed a critical clause." We investigate. Model did retrieve the right section but hallucinated the summary. Week 6: Fixed hallucination issue. Now retrieval is surfacing irrelevant sections. Week 8: Fixed retrieval. Now generation is technically correct but uses jargon customers don't understand. Week 12: CEO asks "How do we know it's actually working?"
We had no answer.
That's when we stopped and built an eval system. Within 2 weeks of having evals, we:
- Identified 3 major failure modes we didn't know existed
- Quantified that retrieval was the bottleneck (not generation)
- Improved precision from 67% to 89%
- Deployed confidently for the first time
The product that took 12 weeks to flounder took 2 weeks to fix once we could measure it.
Our RAG Eval Framework: The 3-Level System
After 8 RAG deployments, we've standardized on a 3-level evaluation framework:
Level 1: Component Tests (Fast, Cheap, Continuous) Level 2: End-to-End Tests (Moderate, Periodic) Level 3: Production Monitoring (Expensive, Always-On)
Let's break down each level.
Level 1: Component Tests (The Foundation)
RAG systems have distinct components that can fail independently:
- Query Understanding: Did we parse the user's intent correctly?
- Retrieval: Did we fetch relevant chunks?
- Ranking: Are the most relevant chunks at the top?
- Context Construction: Did we format context properly for the LLM?
- Generation: Did the LLM produce a good answer given the context?
Example: Retrieval Tests
For a medical documentation RAG system, here's what our Level 1 retrieval tests look like:
# Test: Retrieval Precision for Medical Queries
def test_retrieval_precision_medication_dosage():
query = "What is the recommended dosage of metformin for type 2 diabetes?"
# Expected: Should retrieve dosage guidelines, contraindications
results = retriever.retrieve(query, top_k=5)
# Assertions
assert len(results) > 0, "No results returned"
assert any("metformin" in r.text.lower() for r in results), "No results mention metformin"
assert any("dosage" in r.text.lower() for r in results), "No results mention dosage"
# Check if top result is actually relevant
top_result = results[0]
assert semantic_similarity(query, top_result.text) > 0.7, "Top result not semantically relevant"
We run hundreds of these tests. They cover:
- Edge cases (typos, abbreviations, medical jargon)
- Multiple retrieval methods (vector, keyword, hybrid)
- Different document types (guidelines, studies, protocols)
- Various query complexity levels
These run on every code change via CI/CD. If retrieval precision drops below 80%, the deploy fails.
Example: Generation Tests
For the generation component:
# Test: No Hallucination Outside Context
def test_no_hallucination():
context = """
Metformin dosage: Start with 500mg twice daily with meals.
Maximum dose: 2,550mg per day.
"""
query = "What is the recommended metformin dosage?"
response = generator.generate(query, context)
# Assert: Response should only use information from context
facts_in_response = extract_medical_facts(response)
facts_in_context = extract_medical_facts(context)
for fact in facts_in_response:
assert fact in facts_in_context or is_rephrasing(fact, facts_in_context), \
f"Hallucinated fact: {fact}"
# Assert: Critical numbers must match exactly
assert "500mg" in response, "Dosage not mentioned"
assert "2,550mg" in response or "2550mg" in response, "Max dose not mentioned"
Key principle: These tests must be fast (<100ms per test) so you can run them constantly.
Level 2: End-to-End Evals (LLM-as-a-Judge)
Level 1 tests catch obvious failures. But RAG quality isn't just about components—it's about the whole system working together.
This is where LLM-as-a-Judge comes in.
Our LLM-as-a-Judge Setup
We follow Hamel Husain's "Critique Shadowing" approach:
Step 1: Create Golden Dataset
We curate 100-200 representative queries covering:
- Common questions (80% of traffic)
- Edge cases (15% of traffic)
- Adversarial queries (5% - things we know break)
For each query, we include:
- The user's question
- The RAG system's response
- The retrieved chunks
- Human judgment (Pass/Fail)
- Human critique explaining why
Example from our legal contract RAG:
| Query | Response | Retrieved Chunks | Pass/Fail | Critique | | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | "Does this contract have a non-compete clause?" | "Yes, Section 7.3 contains a non-compete clause restricting employment with competitors for 2 years post-termination." | Sections 7.3, 7.4, 12.1 | Pass | Correct identification and accurate summary. However, should have mentioned geographic scope (California only) which makes this clause likely unenforceable. | | "What are the termination conditions?" | "The contract can be terminated for cause including breach of confidentiality, failure to perform duties, or mutual agreement." | Sections 9.1, 9.2, 15.4 | Fail | Missed critical termination condition: 30-day notice requirement. Also failed to retrieve Section 9.3 which details cure period. This is a retrieval failure, not generation. |
Step 2: Build LLM Judge Prompt
We use the human critiques as few-shot examples for Claude Sonnet 4:
You are an expert legal contract analyst evaluating RAG system outputs.
For each response, you must:
1. Verify factual accuracy against retrieved chunks
2. Check for completeness (are critical details missing?)
3. Assess clarity and usefulness for a non-lawyer
4. Identify whether failures are retrieval or generation issues
Here are examples of good and bad responses with critiques:
<examples>
[Include 5-10 human-labeled examples with detailed critiques]
</examples>
For the following RAG system output, provide:
- Pass/Fail judgment
- Detailed critique explaining your reasoning
- Root cause (retrieval failure, generation failure, or context construction failure)
Query: {{query}}
Retrieved Chunks: {{chunks}}
RAG Response: {{response}}
Step 3: Measure Agreement
We test the LLM judge against our human-labeled dataset:
- Precision: Of the failures the LLM judge flags, how many are real failures?
- Recall: Of the real failures, how many does the LLM judge catch?
For our legal contract RAG, we achieved:
- Precision: 87% (few false positives)
- Recall: 91% (catches most real failures)
This means we can trust the LLM judge to scale our evaluation beyond what humans can manually review.
Step 4: Run at Regular Intervals
We run LLM-as-a-Judge evaluation:
- Weekly on golden dataset (100-200 examples)
- Before each release on expanded test set (500+ examples)
- After major changes (new embedding model, prompt updates, etc.)
What This Catches That Level 1 Misses
Level 1 tests verify components work. Level 2 tests verify the system produces useful outputs.
Example: Our medical documentation RAG passed all Level 1 tests but failed Level 2 because:
- Retrieval returned relevant chunks
- Generation stayed within context
- But responses used medical jargon that patients couldn't understand
The LLM judge flagged: "Technically accurate but uses terms like 'contraindication' and 'titration' without explanation. Fail."
We updated the prompt to include: "Explain medical terms in plain language." Problem solved.
Level 3: Production Monitoring (The Reality Check)
Evals in development are necessary but not sufficient. Real users do unpredictable things.
What We Monitor in Production
1. Retrieval Metrics (Logged for Every Query)
{
"query_id": "abc123",
"query": "user question",
"retrieved_chunks": [...],
"retrieval_score": 0.85, # Top result similarity
"num_results": 5,
"retrieval_latency_ms": 120
}
We alert if:
retrieval_score < 0.6(indicates bad matches)num_results == 0(nothing found)retrieval_latency_ms > 500ms(too slow)
2. Generation Metrics
{
"query_id": "abc123",
"response": "AI response",
"generation_latency_ms": 1500,
"tokens_generated": 350,
"context_size_tokens": 2000,
"refusal_triggered": false # Safety filter
}
We alert if:
generation_latency_ms > 3000(user experience degrades)tokens_generated > 500(too verbose)refusal_triggered == true(safety filter activated)
3. User Feedback (Thumbs Up/Down)
Every response has thumbs up/down buttons. We:
- Sample 20% of thumbs down responses daily
- Manually review to identify patterns
- Add failures to golden dataset
- Re-run Level 2 evals to confirm fixes
4. LLM-as-a-Judge in Production (Sampled)
For 10% of queries, we run our LLM judge in the background:
- Doesn't block the user
- Flags suspicious responses
- Builds dataset for weekly review
If LLM judge failure rate > 15%, we investigate immediately.
Real Results: Before and After Evals
Let me show you concrete numbers from our portfolio companies.
Case Study 1: Legal Contract Analyzer
Before Evals:
- Weeks 1-12: Ad-hoc testing, vibes-based iteration
- User-reported issues: 23 in first 2 weeks of beta
- Precision: Unknown (we literally didn't measure it)
- Time to fix issues: 2-5 days per issue (couldn't reproduce)
After Implementing Evals (Week 13):
- Week 13-14: Built 3-level eval system
- Discovered 3 major failure modes before users hit them
- Precision: 89% (measured)
- Time to fix issues: Hours (reproducible test cases)
6 Months Later:
- User-reported issues: Down 87%
- Weekly eval pass rate: 94%
- Deployment confidence: Team ships weekly instead of monthly
Case Study 2: Medical Documentation RAG
This system retrieves dosage guidelines, treatment protocols, and contraindications from medical literature.
With Evals From Day 1:
- Week 2: Level 1 tests (retrieval, generation)
- Week 4: Level 2 LLM-as-a-Judge system
- Week 6: Beta with 10 doctors
- Week 8: Production with full monitoring
Results:
- Zero hallucinated medical facts in 2,000+ queries
- 92% thumbs-up rate from doctors
- Caught 3 critical issues via production monitoring before they impacted patients
The difference? We didn't ship until evals were green. And when we shipped, we knew it worked.
Our Eval Development Workflow
Here's our actual process for building a RAG system with evals:
Week 1: Build RAG MVP + Level 1 Tests
- Implement basic RAG pipeline
- Write 20-30 Level 1 tests covering:
- Retrieval precision for common queries
- Generation staying within context
- Handling of edge cases (empty results, typos, etc.)
Goal: Catch obvious failures fast.
Week 2-3: Expand Test Coverage
- Add 50+ more Level 1 tests
- Test retrieval across all document types
- Test generation quality variations
- Run tests on every code change
Goal: Stabilize components before moving to end-to-end.
Week 4: Build Golden Dataset
- Collect 100 representative queries
- Have domain expert label Pass/Fail with critiques
- Identify failure patterns
- Fix most common failures
Goal: Understand what "good" looks like for this use case.
Week 5: Build LLM-as-a-Judge
- Create judge prompt with few-shot examples
- Test against golden dataset
- Iterate until precision > 85%, recall > 85%
Goal: Automate human evaluation at scale.
Week 6+: Production with Monitoring
- Deploy with instrumentation
- Run LLM judge on 10% of queries
- Collect user feedback
- Weekly review of flagged queries
Goal: Continuous improvement based on real usage.
Common Mistakes We've Seen (And Made)
Mistake 1: Testing Only Happy Paths
Early on, we tested queries like "What is X?" But real users ask:
- "What's that thing about the payment terms?" (vague)
- "Explain section 7 but ELI5" (unclear intent)
- "Do we need to worry about indemnification?" (yes/no question requiring nuanced answer)
Fix: Curate adversarial examples. Ask "What queries would break this?"
Mistake 2: Using Generic Metrics
We tried using BLEU, ROUGE, BERTScore to evaluate generations. They told us nothing useful.
A response can have high BLEU score and still:
- Hallucinate facts not in context
- Miss critical information
- Be technically correct but useless to users
Fix: Domain-specific evaluation criteria. For legal RAG: accuracy, completeness, clarity, citations. For medical RAG: factual correctness, safety, plain language.
Mistake 3: Skipping Component-Level Tests
We built one RAG system that went straight to end-to-end evals. When evals failed, we couldn't tell if it was retrieval, ranking, or generation.
Debugging took days because we'd change something, re-run expensive evals, and still not know root cause.
Fix: Always have fast component tests. When end-to-end fails, you can quickly isolate the problem.
Mistake 4: Not Involving Domain Experts
Engineers cannot judge medical documentation quality. We don't know what's important.
Our medical RAG had great eval scores but doctors hated it. Why? We were measuring the wrong things.
Fix: Get domain expert to label your golden dataset. Their critiques teach you what actually matters.
Tools We Use
Our stack for RAG evals:
Level 1 (Component Tests):
pytestfor test orchestration- Custom assertion library for RAG-specific checks
- CI/CD: GitHub Actions runs tests on every PR
Level 2 (LLM-as-a-Judge):
- Claude Sonnet 4 as judge
- Golden dataset stored in Postgres
- Weekly eval jobs via cron
Level 3 (Production Monitoring):
- OpenTelemetry for trace logging
- Metabase for dashboards
- PagerDuty for alerts
Cost: ~$200/month for all eval infrastructure (mostly LLM judge calls).
That's far cheaper than the engineer time we save debugging and the user trust we preserve.
When You DON'T Need Evals
Let me be clear: Not every RAG system needs this level of rigor.
Skip this if:
- Your RAG is for internal tools only (5 users)
- Mistakes have no consequences
- You're still in early prototype phase
- You can manually review 100% of outputs
But if:
- Users depend on accuracy (medical, legal, financial)
- Your RAG will handle >100 queries/day
- You're charging money for it
- Failures damage trust or have compliance implications
Then you need evals. No exceptions.
Start Simple, Then Expand
If you're overwhelmed, start here:
Week 1: Build 10 Level 1 tests
- 5 retrieval tests (does it find relevant chunks?)
- 5 generation tests (does it hallucinate?)
Week 2: Create 20-query golden dataset
- Label Pass/Fail manually
- Write critiques for failures
Week 3: Set up basic monitoring
- Log every query/response
- Track retrieval scores
- Add thumbs up/down buttons
Week 4: Review failures weekly
- Sample 20 failed queries
- Identify patterns
- Add tests to prevent repeat failures
That's it. Four weeks to a foundation that will save you months of pain.
The Truth About RAG Systems
Here's what we've learned after 8 deployments:
RAG systems are not "done" when they demo well. They're done when you can prove they work.
Evals aren't overhead. They're how you move fast sustainably.
Every RAG system we've built with evals from day 1 reached production faster, shipped with higher quality, and required less maintenance than the ones we "moved fast" on and added evals later.
The difference is stark. Evals aren't optional for production RAG systems. They're the foundation.
Want to see how we build RAG systems with evals baked in? Check out our Unified Stack series or reach out to discuss your AI project.
