Introduction to RAG¶

1. Why does this topic exist?¶

Before RAG, building an LLM-powered application that answered questions from your own data was nearly impossible. The state of the world looked like this:

The pain points the industry hit (2022-2023):

A bank wanted to build an internal Q&A bot over 50,000 policy PDFs. They tried fine-tuning GPT-3.5 on those PDFs. Cost: $40,000 per training run. Every time a policy changed, the model had to be retrained. The model still hallucinated policy clauses that didn't exist.
A SaaS company tried adding "ChatGPT for your docs" to their product. They pasted documentation into the system prompt. After 8 documents, they hit the context window. Beyond that — silence.
A legal tech startup needed citations. Their lawyer users would not trust answers without "where did this come from?". The LLM had no way to point at a source.
A customer-support team's bot quoted policies from 2019 in 2024 — the model had no idea time had passed since training.

These four pain points have a common shape:

The LLM is brilliant, but it cannot read what it hasn't been trained on, cannot cite sources, and cannot stay current.

Why the previous approaches failed:

Old approach	Why it broke
Fine-tuning	Slow ($1K-$50K), changes locked into weights, hard to update, no citations, still hallucinates
Prompt stuffing	Limited by context window, expensive per query, can't scale to large corpora
Keyword search + LLM	Misses paraphrases — "feline" vs "cat" treated as different queries
Plain LLM	Knowledge cutoff, hallucinations, no private data, no attribution

The industry needed a way to inject fresh, private, citable knowledge into an LLM at query time, without retraining.

That's RAG.

2. What is it?¶

Simple explanation¶

RAG is "open-book exam" mode for an LLM. Instead of asking the LLM to answer from memory, you hand it the right pages of the textbook first, then ask the question.

Technical explanation¶

Retrieval-Augmented Generation (RAG) is a pattern where an LLM's response is conditioned on retrieved, externally-sourced documents that are dynamically fetched at query time from a vector database (or other knowledge store).

answer = LLM(prompt_template(retrieved_context, user_query))

The retrieved context is not in the model's weights — it lives in an external store and is injected into the prompt per query.

Industry definition¶

The term was coined in the 2020 Meta paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Lewis et al. The paper combined a dense passage retriever with a sequence-to-sequence generator. The pattern has since evolved into the dominant production architecture for LLM apps with grounded answers.

Mental model¶

Three actors:

The Librarian (Retriever) — knows where every document is shelved, finds relevant ones fast.
The Intern (LLM) — brilliant reasoner, but has amnesia about everything not in their training.
The Manager (your application) — orchestrates: ask the librarian, hand pages to the intern, get the answer.

Analogy for beginners¶

Imagine a brilliant medical student taking a complex case exam:

Without RAG: They answer from memory. They may forget recent guidelines, confuse symptoms, or invent citations.
With RAG: They have access to the latest UpToDate articles, can flip to the relevant page, and reference it in their answer.

Same student, dramatically better answers — because they have the right context.

3. How does it work?¶

RAG has two phases, running on different timelines.

Phase 1 — Indexing (one-time, offline)¶

Done once per data update. You preprocess your knowledge base into something searchable.

Raw documents → Document Loader → Text Splitter → Embedding Model → Vector Store

Each step:

Load: Read raw files (PDF, HTML, DB rows) into Document objects.
Split: Break long documents into manageable chunks (200-1500 tokens).
Embed: Convert each chunk to a vector that captures its meaning.
Store: Index the vectors for fast similarity search.

Phase 2 — Query (online, per user request)¶

Done every time a user asks something.

User Query → Embed Query → Vector Search → Top-k Chunks → Build Prompt → LLM → Answer

Each step:

Embed query: Same embedding model that was used during indexing.
Search: Find the k vectors most similar to the query vector.
Fetch: Pull the chunks corresponding to those vectors.
Augment prompt: Insert chunks as context into a prompt template.
Generate: LLM produces the answer, grounded in the context.

Mathematical intuition¶

At the heart of RAG is vector similarity. We embed everything into a high-dimensional space (e.g., 1536 dimensions) where:

\[ \text{similarity}(a, b) \approx \cos\theta = \frac{\vec{a} \cdot \vec{b}}{||\vec{a}||\,||\vec{b}||} \]

The closer two vectors point in the same direction, the more semantically similar the texts. Chapter 5 covers all six distance metrics with math + visualization.

Full query lifecycle (sequence diagram)¶

sequenceDiagram
    actor U as User
    participant APP as Application
    participant EM as Embedding Model
    participant VS as Vector Store
    participant LLM as LLM
    U->>APP: What is our refund policy
    APP->>EM: embed_query
    EM-->>APP: query vector
    APP->>VS: search top-k
    VS-->>APP: ranked chunks
    APP->>LLM: prompt with context + question
    LLM-->>APP: answer text
    APP-->>U: answer with citations

Every production RAG system runs this exact sequence per query.

Fine-tuning vs RAG — the cost comparison¶

	Fine-tuning	RAG
Setup cost	$1,000-$50,000 per run	$50-$500 for initial indexing
Update cost	Full retraining for any change	Re-embed only changed chunks
Time to update	Hours to days	Seconds to minutes
Citations	Impossible	Built in
Hallucination	Still possible	Reduced significantly
Per-query latency	Fast (one LLM call)	Slower (retrieve + LLM)
Per-query cost	Lower input tokens	Higher input tokens (context)
Best for	Style, format, persona	Knowledge injection

For 99% of business use cases, RAG wins. Fine-tuning is for "make Claude sound like our brand" not "teach Claude our policies."

4. Visual Learning¶

High-level architecture¶

flowchart LR
    subgraph IDX[Indexing - offline]
        D[Documents] --> L[Loader]
        L --> S[Splitter]
        S --> E[Embedder]
        E --> VS[Vector Store]
    end
    subgraph QRY[Query - online]
        Q[User Query] --> EQ[Query Embedder]
        EQ --> R[Retriever]
        VS -. provides .-> R
        R --> P[Prompt Builder]
        Q --> P
        P --> LLM[LLM]
        LLM --> A[Answer]
    end

Workflow diagram — routing decisions¶

flowchart LR
    A[Receive query] --> B{Should we retrieve}
    B -->|chitchat| C[Direct LLM call]
    B -->|knowledge question| D[Embed query]
    D --> E[Vector search top-k]
    E --> F[Optional rerank]
    F --> G[Build prompt with context]
    G --> H[LLM generates]
    H --> I[Add citations]
    I --> J[Return to user]

System design representation — production tiers¶

flowchart TD
    subgraph CLIENT[Client tier]
        UI[Web/Mobile UI]
    end
    subgraph API[Application tier]
        APP[RAG Orchestrator]
        CACHE[Response cache]
    end
    subgraph DATA[Data tier]
        VS[Vector Store]
        OBJ[Object storage for raw docs]
        DB[Metadata DB]
    end
    subgraph EXT[External services]
        LLMAPI[LLM API]
        EMBAPI[Embedding API]
    end
    UI --> APP
    APP --> CACHE
    APP --> VS
    APP --> LLMAPI
    APP --> EMBAPI
    VS --> OBJ
    APP --> DB

A production RAG service is just a request/response API with vector store + LLM as its two main backends.

Real-world example — customer support bot¶

A SaaS company indexes:

1,200 help-center articles (HTML)
300 API reference pages (Markdown)
50,000 historical support tickets (Postgres)

User asks: "Why is my webhook failing with status 502?"

flowchart LR
    Q[Why is my webhook failing with 502] --> R[Retrieve top 5]
    R --> R1[Help article: Webhook reliability]
    R --> R2[API ref: 5xx error codes]
    R --> R3[Ticket #4521: webhook intermittent 502s]
    R1 & R2 & R3 --> LLM[LLM synthesizes]
    LLM --> A[Step-by-step debug answer with article links]

Three different sources, one coherent answer, every claim citable.

5. Pros¶

Benefit	Detail
Fresh knowledge	Update the vector store; the LLM instantly "knows". No retraining.
Source attribution	Every chunk carries metadata. Easy to cite.
Private data access	Your PDFs, tickets, codebase — all queryable.
Reduced hallucination	LLM grounded in actual text; less room to invent.
Cost-efficient	Index once, query millions of times. No fine-tuning bill.
Auditable	Every answer traceable — critical for legal, finance, healthcare.
Multi-tenant safe	Metadata filtering isolates customer data.
Composable	Swap LLM, embedder, or vector store independently.
Provider-agnostic	Works with OpenAI, Anthropic, Mistral, local models.
Iterative	Add features (reranker, multi-query) without redesigning.

6. Cons¶

Limitation	Detail
Latency overhead	Retrieval adds 100-500ms per query.
Retrieval can miss	If the retriever doesn't find the right chunk, the LLM has nothing to ground on.
Prompt budget pressure	Top-k chunks consume tokens; less room for system prompt and conversation.
Cost of embeddings	Initial indexing for millions of docs can cost $100s-$1000s.
Index staleness	If docs change but the index doesn't, answers reference old content.
Complexity	More moving parts than plain LLM calls.
Garbage in, garbage out	Bad chunks (junk PDFs, wrong encoding) yield bad answers.
Difficult to evaluate	Needs a metric framework (RAGAS — Chapter 12).
Doesn't fix bad LLMs	A weak model with great retrieval still produces weak answers.

7. Trade-offs¶

What we gain	What we lose
Fresh, citable knowledge	A clean single LLM call
Lower hallucination rate	Higher per-query cost & latency
Instant knowledge updates	More infrastructure to maintain
Multi-source synthesis	More complex debugging
Compliance/auditability	Some accuracy on questions outside the corpus

When to use RAG:

✅ You have private/proprietary documents the LLM hasn't seen.
✅ Knowledge changes often.
✅ Citations or audit trails matter.
✅ You need to swap providers without rewriting your data pipeline.

When to avoid:

❌ Pure reasoning tasks (math, logic puzzles).
❌ Single, static dataset that fits in a fine-tune.
❌ Extreme latency constraints (gaming, real-time bidding).
❌ The corpus is tiny (10 docs) — just stuff them in the system prompt.

8. Real-world Industry Usage¶

OpenAI¶

Custom GPTs are RAG underneath — uploaded files get chunked, embedded, retrieved.
OpenAI Assistants API has built-in "file search" — managed RAG over your files.
Enterprise ChatGPT grounds in customer internal docs without sending them to the public model.

Anthropic¶

Claude Projects ground Claude in customer-provided documents via managed RAG.
Citations API returns the exact source passages — built specifically for RAG-style attribution.

Google¶

Gemini Code Assist retrieves from your codebase before answering.
NotebookLM is consumer RAG — upload sources, ask questions, get cited answers.
Vertex AI Search is enterprise RAG-as-a-Service.

Enterprise companies¶

Bloomberg Terminal AI — RAG over market data and news.
GitHub Copilot Chat — RAG over your repo + Stack Overflow.
Salesforce Einstein — RAG over CRM records.
Notion AI — RAG over your workspace.
JPMorgan's "DocLLM" — RAG over legal/financial filings.

Production use cases by industry¶

Industry	Use case
Customer support	Q&A over help center + tickets
Legal	Contract analysis + clause retrieval
Healthcare	Clinical decision support over UpToDate + EHR
Finance	Earnings analysis over 10-K filings
Code	"How do I do X?" over private codebase
Sales	RFP responses grounded in product docs
HR	"What's our policy on X?" over employee handbook

RAG is the default architecture for any business-facing LLM app in 2026.

9. Interview Questions¶

Beginner¶

What does RAG stand for? — Retrieval-Augmented Generation.
What problem does RAG solve? — Grounding an LLM in fresh, private, citable knowledge without retraining.
What are the two phases of a RAG system? — Indexing (offline) and Query (online).
Why split documents into chunks? — To fit embedding model limits and enable precise retrieval.
What's an embedding? — A fixed-length vector capturing the semantic meaning of text.

Intermediate¶

Walk through a user query end-to-end. — Embed → vector search → top-k → prompt → LLM → answer.
How does RAG reduce hallucination? — Grounding the LLM in actual retrieved text + instructing it to use only that context.
When choose fine-tuning over RAG? — When the goal is style/format change, not knowledge injection.
What metrics measure retrieval quality? — Recall@k, MRR (Mean Reciprocal Rank), nDCG.
How handle stale data? — Versioned indexes, incremental re-ingestion, time-decayed metadata.

Advanced¶

Question spans multiple sources — how? — Multi-Query Retriever, ensemble retriever, or agentic RAG with separate tool-retrievers.
Dense vs sparse retrieval? — Dense = embeddings (semantic); sparse = BM25/TF-IDF (keyword). Hybrid combines via RRF.
RAG retrieves right docs, gives wrong answers. Diagnose. — Likely prompt issue (not instructed to stick to context) OR chunks too large OR weak LLM.
How evaluate a RAG system? — RAGAS: Faithfulness, Answer Relevance, Context Precision, Context Recall.
Chunking-vs-context trade-off? — Small chunks = precise match, less context. Large = more context, imprecise. Parent-Document Retriever solves this.

System design¶

Design RAG for 1M help articles, 5 languages. — Cover: chunking, multilingual embedder, vector store with filters, metadata schema, language routing, caching, eval pipeline.
Architect multi-tenant RAG. — Per-tenant collections OR shared store with tenant_id filter; API-gateway auth; rate limiting; cost attribution.
Indexing pipeline for streaming data? — Kafka → consumer → splitter → embedder → upsert; debounce; tombstones for deletes.

Architecture questions¶

Where do you cache? — Embedding layer (same query → cached vector), retrieval layer (query → top-k), LLM layer (context + query → answer).
PII handling? — Mask at ingestion (NER + redaction); store originals access-controlled; unmask only for authorized users.

10. Common Mistakes¶

What beginners do wrong¶

❌ Embedding entire documents instead of chunks. One vector can't represent 50 pages.
❌ Mismatched embedders. Built index with model A, querying with model B → vectors live in different spaces.
❌ No temperature=0 for retrieval-grounded answers. Higher temperature = more invention.
❌ Forgetting to instruct the LLM to use ONLY the context.
❌ Treating top-1 as the answer. Retrievers are rough; pass k≥3 to the LLM.

What production teams do wrong¶

❌ Skipping evaluation. Shipping RAG without test sets = flying blind.
❌ No metadata filtering. Multi-tenant data leaks happen here.
❌ No retrieval logging. When the answer's wrong, can't see what was retrieved.
❌ Re-embedding everything on each deploy. Idempotency matters — use document IDs.
❌ No fallback for retrieval failures. Vector store down → 500s; should degrade gracefully.
❌ Letting the prompt grow unbounded. Truncate intelligently when top-k exceeds budget.

How to avoid failures¶

Build evaluation FIRST. Even 20 questions + ground truths > nothing.
Log every (query, retrieved_chunks, answer) tuple to LangSmith.
Version your index — tag with embedder + splitter config.
Treat the LLM's answer as untrusted input — validate before high-stakes use.
Set up cost alerts. RAG costs scale with traffic and prompt size.

11. Best Practices¶

Industry standards¶

Always use a Pipeline (LangChain Runnable, LangGraph node) rather than ad-hoc glue code.
Decouple indexing from querying. Different schedules, different deploys.
Track index versions. When splitter or embedder changes, build index-v2, dual-run, then switch.
Use with_structured_output() for post-LLM parsing. Never regex-parse JSON.

Production recommendations¶

Cache aggressively. Embeddings of common queries; LLM outputs of identical contexts.
Stream the LLM response. Hide retrieval latency by streaming first token quickly.
Use a managed vector store unless you have the team to operate one (Pinecone, Weaviate Cloud, Vertex AI Search).
Run RAGAS in CI. Block deploys that regress key metrics.
Monitor recall@k weekly. Retrieval quality drifts as content changes.

Optimization techniques¶

Hybrid retrieval (dense + BM25) → +10-20% recall over either alone.
Reranking with a cross-encoder → +20-30% precision over pure vector retrieval.
Query expansion (Multi-Query, HyDE) → +10-15% recall on vague queries.
Parent-Document Retriever when small chunks match precisely but you need surrounding context.

12. Evolution Story¶

The story we tell over the next 11 chapters.

flowchart LR
    A[Static LLM<br/>knowledge cutoff, hallucinations]
    A --> B[Fine-tuning<br/>expensive, locked-in]
    B --> C[Prompt stuffing<br/>context window limit]
    C --> D[Keyword search<br/>misses paraphrases]
    D --> E[Dense embeddings<br/>vectors capture meaning]
    E --> F[Vector databases<br/>fast search at scale]
    F --> G[Fixed chunking<br/>splits mid-sentence]
    G --> H[Recursive/Semantic chunking<br/>preserves boundaries]
    H --> I[Hybrid retrieval<br/>dense + sparse]
    I --> J[Reranking<br/>cross-encoder precision]
    J --> K[Query expansion<br/>multi-query, HyDE]
    K --> L[Agentic RAG<br/>LLM picks tool]
    L --> M[Graph RAG<br/>relationships]
    M --> N[RAGAS<br/>measurable quality]

Each step solves the previous step's limitation. By the end of the course, you'll see why every advanced pattern exists — it's not "extra tooling", it's the answer to a specific real-world failure.

What's next: We start the indexing pipeline. To feed the system data, we need a Document Loader — the bridge between raw files (PDFs, websites, databases) and LangChain's Document shape.

Practice¶

What does this print?

Expected: RAG

full_name = "Retrieval Augmented Generation"
print("".join(w[0] for w in full_name.split()))

Add the fourth LLM problem RAG solves (currently only three)

Expected: 4

rag_solves = [
    "knowledge_cutoff",
    "hallucinations",
    "no_source_attribution",
    # bug: missing "no_private_data_access"
]
print(len(rag_solves))

Quiz — Quick check¶

What you remember

Q1. Which is NOT a problem RAG solves?

Knowledge cutoff
Hallucinations
Lack of source citations
Slow inference speed

Why: RAG actually adds latency (retrieval + bigger prompts). It targets freshness, hallucinations, attribution.

Q2. When does indexing run?

Once per data update, offline
On every user query
Only at app startup
Whenever the LLM hallucinates

Why: Indexing is expensive and offline. Query-time hits the already-built index.

Q3. Fine-tuning vs RAG — when is each preferred?

Fine-tune for style/format; RAG for knowledge injection
Always fine-tune
Always RAG
They're identical

Why: Fine-tuning changes how the model behaves; RAG changes what it knows. For frequently-updating knowledge, RAG wins on every dimension (cost, time, citations).

Common doubts¶

Is RAG just 'prompt engineering with extra steps'?

Sort of — but the "extra steps" do work prompt engineering can't. You can't fit a million documents into a prompt; RAG selects the right few. Plus the retrieval layer lets you update knowledge without code changes.

Will fine-tuning disappear because of RAG?

No. They're complementary. RAG injects facts; fine-tuning shapes behavior (tone, format, output structure). Many production apps use both: fine-tune for style + RAG for knowledge.

Is RAG just a temporary hack until context windows are infinite?

Even with infinite context, you'd still want RAG: cost (sending 1M tokens per query is expensive), latency (LLMs slow with long context), and "lost in the middle" effects (LLMs attend less to middle context). RAG is here to stay.

→ Document Loaders

Old approach	Why it broke
Fine-tuning	Slow (\(1K-\)50K), changes locked into weights, hard to update, no citations, still hallucinates
Prompt stuffing	Limited by context window, expensive per query, can't scale to large corpora
Keyword search + LLM	Misses paraphrases — "feline" vs "cat" treated as different queries
Plain LLM	Knowledge cutoff, hallucinations, no private data, no attribution

	Fine-tuning	RAG
Setup cost	\(1,000-\)50,000 per run	\(50-\)500 for initial indexing
Update cost	Full retraining for any change	Re-embed only changed chunks
Time to update	Hours to days	Seconds to minutes
Citations	Impossible	Built in
Hallucination	Still possible	Reduced significantly
Per-query latency	Fast (one LLM call)	Slower (retrieve + LLM)
Per-query cost	Lower input tokens	Higher input tokens (context)
Best for	Style, format, persona	Knowledge injection