RAGAS — Evaluating RAG Systems¶

1. Why does this topic exist?¶

We've built every RAG variant. Now we have to answer: is it any good?

Industry pain example: A team shipped Multi-Query + Reranking + Parent-Doc, convinced it was better. A user complained: "It used to find my docs, now it doesn't." They had no evaluation framework — no way to verify the user was right, no way to identify the regression. Six weeks of debugging before they admitted Multi-Query had degraded retrieval for their specific corpus.

Without evaluation, every change is a guess. RAGAS (Retrieval Augmented Generation Assessment) is the standard framework for measuring RAG quality scientifically.

The four hard questions evaluation must answer:

Does retrieval find the right docs? (Context Recall)
Are retrieved docs relevant? (Context Precision)
Is the answer grounded in retrieved docs? (Faithfulness)
Does the answer address the question? (Answer Relevance)

Each is a different failure mode. Each maps to a different component fix.

2. What is it?¶

Simple explanation¶

RAGAS gives you 4 numbers per RAG system. Higher is better. When one drops, you know which component is broken.

Technical explanation¶

RAGAS is an open-source library that evaluates RAG outputs by treating an LLM as a judge. For each test question, it scores:

Faithfulness: fraction of answer claims supported by context.
Answer Relevance: similarity between original question and questions inferred from the answer.
Context Precision: fraction of retrieved chunks that are relevant.
Context Recall: fraction of ground-truth answer claims supported by context.

Industry definition¶

RAGAS originated from a 2023 paper by Exploding Gradients. It's now used by hundreds of production teams and ships as the de facto evaluation framework for LangChain/LlamaIndex applications.

Mental model¶

Think of RAGAS as automated peer review for your RAG system. A senior engineer (LLM judge) reads each answer + the context it was generated from, and grades on four axes.

Analogy¶

Code coverage in software — without it, you don't know which code paths your tests exercise. Without RAGAS, you don't know which RAG components are working.

3. How does it work?¶

The four metrics — math + intuition¶

Faithfulness — "Did the LLM hallucinate?"¶

\[ \text{Faithfulness} = \frac{\#\text{ claims in answer supported by context}}{\#\text{ total claims in answer}} \]

Process:

LLM extracts atomic claims from the answer.
For each claim, LLM checks: is this supported by the retrieved context?
Score is the supported ratio.

Score range: $[0, 1]$. Below 0.7 = serious hallucination problem.

Answer Relevance — "Did the LLM answer the question?"¶

\[ \text{Answer Relevance} = \frac{1}{N} \sum_{i=1}^{N} \cos(\vec{q_{\text{orig}}}, \vec{q_i}) \]

Process:

LLM generates N candidate questions from the answer.
Embed original question and generated questions.
Average cosine similarity.

Intuition: an off-topic answer produces off-topic generated questions → low similarity to the original. A well-targeted answer produces questions similar to the original.

Context Precision@K — "Are retrieved chunks relevant?"¶

\[ \text{Context Precision@K} = \frac{\sum_{k=1}^{K} \text{Precision}@k \cdot v_k}{\#\text{ relevant in top-K}} \]

Where $v_k = 1$ if chunk at rank $k$ is relevant, 0 otherwise.

Penalizes irrelevant chunks AND wrong ordering. Below 0.5 = retrieval returns noise.

Context Recall — "Did retrieval find ALL relevant info?"¶

\[ \text{Context Recall} = \frac{\#\text{ ground-truth claims supported by retrieved context}}{\#\text{ total ground-truth claims}} \]

Process:

Use the ground-truth answer (you must label it).
Extract claims from ground truth.
Check each claim against retrieved context.

Requires labeled ground truth. Below 0.6 = retrieval is missing facts.

Mapping metric drops to component fixes¶

flowchart TD
    LOW[Low metric] --> F[Low Faithfulness]
    LOW --> R[Low Answer Relevance]
    LOW --> P[Low Context Precision]
    LOW --> C[Low Context Recall]
    F --> F1[Fix prompt: tell LLM to stick to context, lower temperature]
    R --> R1[Fix synthesis prompt: make question clearer]
    P --> P1[Smaller chunks, MMR, reranking]
    C --> C1[Bigger k, Multi-Query, HyDE, hybrid retrieval]

This map is the most important thing in this chapter. Score drops point you directly to a component fix.

Building an evaluation set¶

test_set = [
    {
        "question": "What is the refund window for our products?",
        "ground_truth": "The refund window is 30 days from purchase.",
    },
    {
        "question": "Which plan includes priority support?",
        "ground_truth": "Enterprise plan includes 24/7 priority support.",
    },
    # 50-100 questions across topics, difficulties
]

Running RAGAS¶

from ragas import evaluate
from ragas.metrics import (
    faithfulness, answer_relevancy,
    context_precision, context_recall,
)
from datasets import Dataset

# 1. Run your RAG pipeline on each question
rows = []
for item in test_set:
    docs = retriever.invoke(item["question"])
    answer = rag_chain.invoke(item["question"])
    rows.append({
        "question": item["question"],
        "answer": answer,
        "contexts": [d.page_content for d in docs],
        "ground_truth": item["ground_truth"],
    })

# 2. Evaluate
ds = Dataset.from_list(rows)
scores = evaluate(
    dataset=ds,
    metrics=[faithfulness, answer_relevancy,
             context_precision, context_recall],
)

print(scores)
# {'faithfulness': 0.84, 'answer_relevancy': 0.91,
#  'context_precision': 0.78, 'context_recall': 0.82}

4. Visual Learning¶

Architecture — evaluation pipeline¶

flowchart LR
    TS[Test set Q + ground truth] --> RAG[Your RAG system]
    RAG --> R[Q + answer + retrieved contexts]
    R --> RAGAS[RAGAS evaluator]
    RAGAS --> JUDGE[LLM-as-judge per metric]
    JUDGE --> S[Four scores]
    S --> DASH[Dashboard or CI]

Sequence — single evaluation run¶

sequenceDiagram
    participant CI as CI/CD
    participant TS as Test Set
    participant RAG as RAG Pipeline
    participant RAGAS
    participant LLM as Judge LLM
    CI->>TS: load
    loop For each question
        CI->>RAG: ask question
        RAG-->>CI: answer + contexts
    end
    CI->>RAGAS: evaluate(dataset)
    loop For each metric
        RAGAS->>LLM: judge
        LLM-->>RAGAS: per-row score
    end
    RAGAS-->>CI: aggregate scores
    CI->>CI: pass / fail vs baseline

Iteration loop¶

flowchart LR
    BASE[Build baseline RAG] --> M[Measure with RAGAS]
    M --> H[Hypothesize fix]
    H --> CHG[Change ONE thing]
    CHG --> M2[Re-measure]
    M2 -->|better| KEEP[Keep change]
    M2 -->|worse or flat| REVERT[Revert]
    KEEP --> H
    REVERT --> H

One change at a time. Multiple changes obscure causality.

Real-world example — customer support RAG¶

Iteration	Change	Faithfulness	Ans Rel	Ctx Prec	Ctx Rec
0	Baseline (dense retrieval, k=4)	0.72	0.81	0.58	0.65
1	Add BM25 (hybrid)	0.74	0.82	0.62	0.79 ↑
2	Add Cohere Rerank	0.81	0.85	0.84 ↑	0.78
3	Add Contextual Compression	0.88 ↑	0.86	0.91 ↑	0.77
4	Temperature=0 + "use only context" prompt	0.94 ↑	0.87	0.91	0.77

Each step shows which metric improved — and which didn't.

5-7. Pros / Cons / Trade-offs¶

Pros¶

Quantitative — replaces "feels better" with numbers.
Component-specific — bad metric points to which part to fix.
CI-friendly — block deploys that regress.
LLM-as-judge — scales beyond manual review.

Cons¶

Stochastic judge — multiple runs differ slightly. Average or use seed.
Costs $$$ — each test question = multiple LLM calls. 100 questions × 4 metrics ≈ $1-5.
Requires ground truth — building a quality test set takes time.
Judge LLM bias — may share biases with the system being tested.

Trade-offs¶

Choice	Trade-off
Big test set	Better signal, more cost
Strong judge model	Better quality, more cost
Run on CI vs ad-hoc	Catches regressions, more spend
Add all metrics vs subset	Comprehensive, expensive

8. Real-world Industry Usage¶

OpenAI¶

Internal evals use RAGAS-style metrics adapted for their use cases.
Their published RAG benchmarks use the four standard metrics.

Anthropic¶

Contextual Retrieval paper evaluates with RAGAS-compatible metrics; reports 67% reduction in retrieval failures.

Enterprise¶

LangSmith integrates RAGAS for one-click evaluation.
LlamaIndex ships its own evaluation suite mirroring RAGAS.
Production teams at Notion, GitHub, Stripe run RAGAS in CI.

Production patterns¶

Pattern	Where
Run RAGAS on each PR	RAG-heavy SaaS
Threshold-based deploy gates	Mature pipelines
Trend monitoring (weekly)	Catch data drift
Per-tenant RAGAS for multi-tenant	Cohort quality

9. Interview Questions¶

Beginner¶

Why evaluate RAG? — Without metrics, every improvement is a guess.
What are the four core RAGAS metrics? — Faithfulness, Answer Relevance, Context Precision, Context Recall.
Which metric catches hallucinations? — Faithfulness.

Intermediate¶

Low Context Recall means what? — Retrieval is missing relevant chunks → expand k, add Multi-Query / HyDE / hybrid.
Low Faithfulness? — LLM is inventing — fix prompt, lower temperature.
What's "LLM-as-judge"? — Use an LLM to score outputs against rubrics. Cheaper and more scalable than human review.

Advanced¶

Why is Faithfulness not enough? — A model that always says "I don't know" has perfect Faithfulness (no claims) but zero usefulness. Combine with Answer Relevance.
How handle judge LLM bias? — Use a stronger model as judge than the one being evaluated; run multiple rounds; sample-check against human labels.
Custom metric for your use case? — Define a rubric; use RAGASCustomMetric or write your own LLM judge with a prompt + structured output.

System design¶

Build CI pipeline for RAG. — Test set in Git; on every PR, run pipeline + RAGAS; compare vs baseline (last green build); fail build if metrics drop >5%.
Diagnose: Faithfulness dropped from 0.85 to 0.70 between two builds. — Diff what changed (prompt, model, retrieval). Run RAGAS on per-question level; find which questions newly fail; look at retrieved contexts vs answers.

10. Common Mistakes¶

❌ Tiny test set (5 questions) → noise dominates signal.
❌ Test set mirrors training data → falsely high scores.
❌ One RAGAS run = ground truth (it's stochastic).
❌ Optimizing for Faithfulness alone → model refuses to answer.
❌ Manual evaluation without rubric → reviewers disagree.

11. Best Practices¶

50-100 questions minimum for stable signal.
Cover the real-world distribution — easy, hard, edge cases.
Stronger judge than system (gpt-4o judging gpt-4o-mini).
Run RAGAS 3x and average for stability.
CI gate: block if any metric drops >5% from baseline.
Add custom metrics for domain-specific needs (citation correctness, fact accuracy).

12. Evolution Story — closing the course¶

flowchart LR
    A[Linear RAG built] --> B[Manually eyeball outputs<br/>subjective]
    B --> C[BLEU / ROUGE<br/>doesn't capture meaning]
    C --> D[Human review<br/>doesn't scale]
    D --> E[RAGAS LLM-as-judge<br/>scales + metric-specific]
    E --> F[CI-integrated RAG eval<br/>regression detection]
    F --> G[Custom metrics<br/>domain-specific quality]

Where we end: With evaluation, you can iteratively improve RAG with confidence. Without it, you're guessing.

The full course in one map¶

flowchart LR
    A[1. Intro: Why RAG] --> B[2. Loaders]
    B --> C[3. Chunking: 7 strategies]
    C --> D[4. Embeddings + 6 distance metrics]
    D --> E[5. Vector DBs: 7 options]
    E --> F[6. Retrievers: Dense + Sparse + Hybrid]
    F --> G[7. Advanced: Multi-Query, Self-Query, Compression, Parent-Doc, Reranking]
    G --> H[8. RAG Fusion + RRF]
    H --> I[9. HyDE]
    I --> J[10. Agentic + 5 variants]
    J --> K[11. Graph RAG]
    K --> L[12. RAGAS Evaluation]

You now have: - A complete mental model of the RAG stack. - Practical code patterns for every component. - An evaluation framework to ship with confidence. - The vocabulary to discuss RAG at interview and production level.

You're ready to design, build, optimize, debug, and ship a production-grade RAG system from scratch.

Practice¶

What does this print?

Expected: 4

metrics = ["faithfulness", "answer_relevance", "context_precision", "context_recall"]
print(len(metrics))

Compute Faithfulness correctly (supported / total claims)

Expected: 0.75

total_claims = 4
supported = 3
bad = supported / 5             # bug
print(round(supported / total_claims, 2))

Quiz — Quick check¶

What you remember

Q1. Which metric catches hallucinations?

Faithfulness
Context Precision
Answer Relevance
Context Recall

Q2. Low Context Recall → which component to fix?

Retrieval (bigger k, Multi-Query, HyDE, hybrid)
LLM
Embedding model
Chunking only

Q3. Right iteration loop?

Measure baseline → change ONE thing → re-measure → keep or revert
Change many things then measure
Measure once, ship
No need to measure

Common doubts¶

How big should my test set be?

50-100 for initial. 500+ for production-grade monitoring. Quality > quantity — cover the real distribution.

Which LLM as judge?

Stronger than the system being tested. If your RAG uses gpt-4o-mini, use gpt-4o or Claude Opus as judge.

Should RAGAS run in CI?

Yes, on a fixed test set. Block PRs that regress metrics by >5%. This is the missing CI step in most LLM apps.

← Back to Advanced RAG home