Text Embeddings & Similarity Search¶

1. Why does this topic exist?¶

After chunking, we have ~thousands of text chunks. The retrieval problem: given a user query, find the chunks most relevant to it.

The old solution — keyword search (TF-IDF, BM25) — searches for exact word overlap:

Query: "feline pets that purr"
Result: only docs containing "feline" AND "purr"
Miss: docs about "cats" (semantically identical but no word match)

Industry pain example: A customer support team built keyword search over their help articles. Users typed "my software is acting up" → no match. Articles titled "troubleshooting application errors" sat unread. The team manually maintained synonym dictionaries — endless. They needed meaning-based matching.

Embeddings solve this: map every chunk and every query into a high-dimensional space where semantic similarity = geometric proximity.

"Cats" and "felines" become nearby points in vector space, despite zero word overlap.

This is the foundational shift that made modern RAG possible.

2. What is it?¶

Simple explanation¶

An embedding is a way to turn text into a list of numbers, where similar texts produce similar lists.

Technical explanation¶

An embedding model is a neural network that takes text input and outputs a fixed-length vector $\vec{v} \in \mathbb{R}^d$ (e.g., $d=1536$). The training objective ensures semantically similar texts produce vectors with small geometric distance.

"I love dogs"          → [0.21, 0.84, ..., 0.12]  (1536 dims)
"Canines are my favs"  → [0.20, 0.83, ..., 0.13]  ← nearly same vector
"Pizza is delicious"   → [0.93, 0.12, ..., 0.78]  ← far away

Industry definition¶

The pattern goes back to word2vec (2013) → GloVe (2014) → BERT sentence embeddings (2019) → OpenAI ada / text-embedding-3 (2022-2023). Today, "embedding" specifically means a dense vector from a transformer model.

Mental model¶

Imagine a map of the world where every city is a piece of text. Cities about "cats" cluster in one region. Cities about "finance" cluster in another. Cosine similarity is the bird's-eye distance between two cities.

Analogy¶

Think of GPS coordinates. Two cities with similar GPS coordinates (e.g., latitude/longitude near 19°N, 73°E) are physically close, regardless of their names. Embeddings give every piece of text a "semantic GPS coordinate."

3. How does it work?¶

Dense vectors¶

The standard form. Every dimension is a real number, most are non-zero.

[0.21, -0.05, 0.84, ..., 0.12]   # 1536 dimensions of meaning

Sparse vectors¶

A different shape used by older models (TF-IDF, BM25). Each dimension is a vocabulary word; most are zero.

[0, 0, 0, ..., 0.95 (for "cat"), 0, ..., 0.83 (for "pet"), 0, 0, ...]   # vocab-sized

Dense vs sparse:

	Dense	Sparse
Dimensions	Fixed (~1500)	Vocab size (~30K-100K)
Most values	Non-zero	Zero
What it captures	Semantics	Keyword frequency
Storage	Compact	Compressible (sparse format)
Use	Modern RAG	BM25, hybrid retrieval

Modern RAG uses BOTH — dense for semantic matching, sparse (BM25) for keyword matching, combined via hybrid retrieval (Chapter 6).

High-dimensional space — why so many dimensions?¶

Concept	Why dimensions needed
Topic (sports vs cooking)	2-3 dims
Subtopic (basketball vs cricket)	5-10 dims
Sentiment (positive vs negative)	1-2 dims
Style (formal vs casual)	1-2 dims
Many such latent factors	100s-1000s

Each dimension captures a latent factor. The model learns these from billions of examples during pre-training.

Why embeddings "work"¶

Modern embedders are trained with contrastive learning:

Positive pairs (semantically similar) → pulled together in vector space.
Negative pairs (unrelated) → pushed apart.

After training on hundreds of millions of pairs:

\[ \text{sim}(\vec{a}, \vec{b}) \approx \text{semantic\_similarity}(a, b) \]

Vector indexing — making search fast¶

A 1M-vector store can't compare against every vector per query. Approximate Nearest Neighbor (ANN) algorithms like HNSW, IVF, and PQ make search O(log n) instead of O(n). We deep-dive in Chapter 5.

The major embedding models — chosen by trade-off¶

Model	Dimensions	Cost (1M tokens)	Best for
`text-embedding-3-small` (OpenAI)	1536	$0.02	General default
`text-embedding-3-large` (OpenAI)	3072	$0.13	High-quality enterprise
`voyage-3` (Voyage AI)	1024	$0.06	RAG-optimized
`cohere-embed-v3` (Cohere)	1024	$0.10	Multilingual + reranker bundle
`all-MiniLM-L6-v2` (local)	384	Free	Local, fast, decent quality
`bge-large-en-v1.5` (local)	1024	Free	Local, top-tier open-source
`jina-embeddings-v3` (Jina)	1024 (variable)	$0.02	Long context, late chunking

Asymmetric models — query vs document embedding¶

Some models embed queries and docs differently because they're trained on (query, doc) pairs to maximize retrieval quality.

emb = OpenAIEmbeddings()
query_vec = emb.embed_query("What is RAG?")           # asymmetric path
doc_vecs  = emb.embed_documents(["RAG is...", "..."]) # different path

For symmetric models, both paths produce identical output. Always call the right one — costs nothing, helps when it matters.

Similarity Search — the math¶

This is the most important section in the entire course. You cannot reason about retrieval quality without understanding these.

Metric 1: Cosine Similarity ⭐ (the RAG default)¶

Formula:

\[ \cos(\theta) = \frac{\vec{a} \cdot \vec{b}}{||\vec{a}||\,||\vec{b}||} \]

Range: $[-1, 1]$. Higher = more similar.

Intuition: the cosine of the angle between two vectors. Ignores magnitude — only direction matters.

flowchart LR
    A[Vector A] --> O[Origin]
    B[Vector B] --> O
    O --> ANGLE[angle theta]

Visualization:

Cosine	Angle	Meaning
1.0	0°	Identical direction → same meaning
0.7	45°	Strongly related
0.0	90°	Unrelated
-1.0	180°	Opposite (rare with text embeddings)

Python:

import numpy as np

def cosine(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

a = np.array([1, 0.5, 0.2])
b = np.array([0.9, 0.6, 0.25])
c = np.array([-0.3, 0.1, 0.9])

print(cosine(a, b))   # 0.99 — similar direction
print(cosine(a, c))   # 0.04 — unrelated

Advantages: - Length-invariant (a long doc vs short doc still compares fairly). - Range bounded → easy to threshold. - Standard across vector stores.

Limitations: - Ignores absolute magnitude (sometimes useful info).

When to use: Default for text embeddings. OpenAI, Cohere, Voyage all use cosine. Always pick this unless you have a specific reason.

Metric 2: Dot Product¶

Formula:

\[ \vec{a} \cdot \vec{b} = \sum_{i=1}^{d} a_i \cdot b_i \]

Range: unbounded. Higher = more similar (if vectors normalized).

Intuition: sum of element-wise products. Sensitive to both direction AND magnitude.

Python:

import numpy as np
a = np.array([1, 2, 3])
b = np.array([4, 5, 6])
print(np.dot(a, b))   # 1*4 + 2*5 + 3*6 = 32

When dot product = cosine: when vectors are normalized to unit length (||a|| = ||b|| = 1). Then dot(a, b) = cos(angle).

Advantages: - Faster than cosine (no normalization at query time if vectors are pre-normalized). - Most vector stores internally use dot product on pre-normalized vectors.

Limitations: - Sensitive to magnitude unless normalized. - Range unbounded → thresholds depend on data.

When to use: If your embedder produces pre-normalized vectors (most modern ones do — text-embedding-3-* returns unit vectors), dot product is mathematically equivalent to cosine and slightly faster. Pinecone, FAISS, and Qdrant let you pick.

Metric 3: Euclidean Distance (L2)¶

Formula:

\[ d(\vec{a}, \vec{b}) = \sqrt{\sum_{i=1}^{d} (a_i - b_i)^2} \]

Range: $[0, \infty)$. Lower = more similar.

Intuition: Straight-line distance between two points in space.

flowchart LR
    A[Vector A 1 2] --> S[Squared diffs]
    B[Vector B 4 6] --> S
    S --> RES[d = sqrt 9 + 16 = 5]

Python:

import numpy as np
a = np.array([1, 2])
b = np.array([4, 6])
print(np.linalg.norm(a - b))   # 5.0

Advantages: - Intuitive — "real" distance. - Works regardless of normalization.

Limitations: - Sensitive to magnitude (a 2× longer vector is "far" from a 1× version of the same direction). - Bad for high-dimensional spaces ("curse of dimensionality" — all points become equidistant).

When to use: When embeddings aren't normalized AND magnitude is meaningful. Rare in modern text embeddings; common in numerical features.

Relationship to cosine: For unit-normalized vectors, $d_{\text{euclidean}}^2 = 2(1 - \cos\theta)$ — they're monotonically related. So for normalized vectors, ranking by Euclidean = ranking by cosine.

Metric 4: Manhattan Distance (L1, Taxicab)¶

Formula:

\[ d(\vec{a}, \vec{b}) = \sum_{i=1}^{d} |a_i - b_i| \]

Intuition: Distance if you can only move along axes (like a taxi in Manhattan's grid).

flowchart LR
    A[A: 0 0] --> R[Manhattan path: right then up]
    R --> B[B: 4 3]
    A --> E[Euclidean: diagonal]
    E --> B

For A = (0,0), B = (4,3): - Manhattan = |4-0| + |3-0| = 7 - Euclidean = √(16+9) = 5

Python:

import numpy as np
a = np.array([0, 0])
b = np.array([4, 3])
print(np.sum(np.abs(a - b)))   # 7

Advantages: - Less sensitive to outliers than Euclidean. - Faster to compute (no squaring/sqrt).

Limitations: - Less geometrically meaningful for continuous embeddings.

When to use: Rare for text embeddings. More common for tabular features with mixed scales, or for sparse vectors where coordinate-wise differences matter.

Metric 5: Hamming Distance¶

Formula:

\[ d_H(\vec{a}, \vec{b}) = \sum_{i=1}^{d} \mathbb{1}[a_i \neq b_i] \]

(Count of positions where they differ.)

Intuition: "How many positions need to change to turn one into the other?"

a = 1 0 1 1 0 1
b = 1 1 1 0 0 1
    diff in positions 1, 3 → Hamming = 2

Python:

a = "101101"
b = "111001"
hamming = sum(c1 != c2 for c1, c2 in zip(a, b))
print(hamming)   # 2

Advantages: - Extremely fast on binary vectors (CPU XOR + popcount). - Used in binary embeddings (1-bit per dimension) for ultra-low-storage RAG.

Limitations: - Only works for binary / discrete vectors. - Not for continuous float embeddings.

When to use: Binary-quantized embeddings for massive scale. Cohere and others now offer binary embeddings — 32× smaller storage, search via Hamming. Quality drops ~5%, storage drops 96%. Used by some search engines at extreme scale.

Metric 6: Jaccard Similarity¶

Formula:

\[ J(A, B) = \frac{|A \cap B|}{|A \cup B|} \]

(Intersection over union for sets.)

Intuition: "How much overlap, normalized by combined size?"

A = {"cat", "dog", "fish"}
B = {"cat", "dog", "bird"}
J = |{"cat", "dog"}| / |{"cat", "dog", "fish", "bird"}|
  = 2 / 4
  = 0.5

Python:

def jaccard(a, b):
    a, b = set(a), set(b)
    return len(a & b) / len(a | b)

print(jaccard(["cat","dog","fish"], ["cat","dog","bird"]))   # 0.5

Advantages: - Set-based, ignores order and duplicates. - Natural for tag sets, n-gram comparisons, sparse keyword vectors.

Limitations: - Not for dense continuous embeddings. - Sensitive to set sizes (big sets dilute).

When to use: Deduplication ("is this document a near-duplicate?"), tag matching, MinHash-based document similarity. Not for transformer embeddings.

Choosing the right metric — quick decision tree¶

flowchart TD
    A[What kind of vector] --> B{Type}
    B -->|Dense float modern embedder| C[Cosine or Dot pre-normalized]
    B -->|Sparse BM25/TF-IDF| D[Dot product or BM25 score]
    B -->|Binary quantized| E[Hamming]
    B -->|Set of tokens/n-grams| F[Jaccard]
    B -->|Tabular features| G[Euclidean or Manhattan]

For modern text RAG: cosine (or equivalent: pre-normalized dot product). Pinecone, FAISS, Qdrant, Weaviate all default to it.

4. Visual Learning¶

Architecture — embedding-based retrieval¶

flowchart LR
    subgraph IDX[Indexing]
        D[Chunks] --> EM[Embedder]
        EM --> V[Vectors]
        V --> VS[Vector Store with ANN index]
    end
    subgraph QRY[Query]
        Q[User Query] --> EQ[Embedder]
        EQ --> QV[Query vector]
        QV --> SS[Similarity search cosine]
        VS -. lookup .-> SS
        SS --> TK[Top-k chunks]
    end

Workflow — query → ranked chunks¶

flowchart LR
    A[Query string] --> B[Embed query]
    B --> C[Send to vector store]
    C --> D[ANN search find candidate vectors]
    D --> E[Compute cosine similarity for candidates]
    E --> F[Sort by similarity]
    F --> G[Return top-k chunks]

Sequence — semantic search in production¶

sequenceDiagram
    actor U as User
    participant API
    participant Cache
    participant Embedder
    participant VS as Vector Store
    U->>API: search "tell me about RAG"
    API->>Cache: get cached query embedding
    Cache-->>API: miss
    API->>Embedder: embed_query
    Embedder-->>API: 1536-dim vector
    API->>Cache: store vector
    API->>VS: knn k=5
    VS-->>API: top-5 chunks with scores
    API-->>U: results

Real-world example — semantic search of help docs¶

flowchart LR
    Q[my app is acting up] --> EQ[Embed query]
    EQ --> QV[Query vector]
    QV --> SEARCH[Cosine similarity search]
    SEARCH --> R1[Troubleshooting application errors 0.84]
    SEARCH --> R2[Common application crashes 0.79]
    SEARCH --> R3[Reset application state 0.72]

Notice: NONE of the retrieved articles contain the words "acting up". They're all semantically related. Keyword search would return nothing.

5. Pros¶

Benefit	Detail
Semantic match	"Cats" matches "felines" without manual synonym dictionaries.
Multilingual	Modern embedders work across languages.
Zero-shot	No training needed — just embed and search.
Scalable	ANN indexes (HNSW, IVF) make 100M-vector search fast.
Composable	Works with any LLM, any vector store.
Cheap at scale	$0.02 per 1M tokens (OpenAI 3-small).

6. Cons¶

Limitation	Detail
Exact terms can miss	Vector match struggles with product codes, named entities ("SKU-A4-12").
Embedding cost	Initial indexing of millions of docs: $100-$1000.
Black box	Hard to debug "why was this retrieved?"
Model lock-in	Re-embed all docs if you change models.
Dimensionality	More dims = better quality + more storage + slower search.
Sensitive to chunking	Bad chunks → bad embeddings → bad retrieval.

7. Trade-offs¶

Decision	Trade-off
Larger embedder	Better quality, 2-4× cost & storage
Local vs API	Privacy + free vs convenience + quality
Higher dim	Quality vs speed + storage
Symmetric vs asymmetric	Simplicity vs slight retrieval gain
Pre-normalized	Faster query vs slight storage

Decision rule:

Start with text-embedding-3-small. Measure. Upgrade only if metrics demand.
For privacy: bge-large-en-v1.5 (local).
For multilingual: cohere-embed-v3-multilingual or text-embedding-3-large.
For long-context: jina-embeddings-v3.

8. Real-world Industry Usage¶

OpenAI¶

ChatGPT memory uses embeddings to retrieve relevant past conversations.
File search in Assistants API: text-embedding-3-large by default.

Anthropic¶

Claude with Contextual Retrieval uses voyage-2 or cohere-embed-v3.
Anthropic's research: embedder choice changes recall by 10-20%.

Google¶

Vertex AI Search uses Google's proprietary embeddings.
NotebookLM uses Gemini-derived embeddings.

Enterprise patterns¶

Use case	Common embedder
General SaaS RAG	`text-embedding-3-small`
Legal / compliance	`text-embedding-3-large` or `voyage-large`
Multilingual customer support	`cohere-embed-v3-multilingual`
On-premise / privacy	`bge-large-en-v1.5` (BAAI)
Code search	`voyage-code-2`, `jina-embeddings-v2-base-code`
Massive scale (>100M chunks)	Binary-quantized embeddings (Cohere, ColBERT)

9. Interview Questions¶

Beginner¶

What's an embedding? — Fixed-length vector representing text meaning.
Why use cosine similarity? — Length-invariant; standard for normalized text embeddings.
Dense vs sparse vectors? — Dense (modern, continuous, ~1500 dims). Sparse (vocab-sized, mostly zeros, e.g., TF-IDF).

Intermediate¶

What does it mean if cosine = 0.95? — Very similar direction → very similar meaning.
Why use embed_query vs embed_documents? — Asymmetric models embed queries and docs differently; right method preserves retrieval quality.
What's the relationship between dot product and cosine? — Identical for unit-normalized vectors.
How do you reduce embedding cost? — Cache; batch; pick smaller model; smaller dims; use OpenAI's dimensions parameter to truncate.

Advanced¶

Why does cosine fail in extremely high dimensions? — Curse of dimensionality — all points become equidistant. Modern embedders combat this with contrastive training.
Binary embeddings — when worth it? — When storage cost dominates and you can tolerate ~5% quality loss (Cohere binary embeddings store 32× less).
How do you handle multilingual queries? — Use a multilingual embedder; OR translate at query time; OR maintain per-language indexes.

System design¶

Design a real-time embedding service for 10K QPS. — Embedder as a service (TGI / Triton); embedding cache (Redis); batch low-priority writes; sharded by language.
A team has 50M documents to embed. Plan the rollout. — Batched parallel jobs (1000 docs/batch); checkpoint-restart; idempotency by content hash; cost: ~$1000 with 3-small. Track progress in a Postgres table.

10. Common Mistakes¶

Beginners¶

❌ Different embedders for indexing vs querying.
❌ Truncating long chunks silently (embedder caps at 8K tokens).
❌ Re-embedding everything for every dev experiment — burns money.
❌ Using Euclidean distance for non-normalized text embeddings.

Production teams¶

❌ Not caching common queries.
❌ Mixing models in one vector store — vectors live in different spaces, results random.
❌ Hard-coding the embedder version in code — can't tell which index was built with what.
❌ Ignoring tokenizer mismatch when switching providers.

How to avoid¶

Always save embedder name + dimension as metadata with the vector store.
Cache with CacheBackedEmbeddings during development.
For production rollouts: build new index in parallel, A/B test, then switch.
Audit chunk lengths before embedding — fail loudly if any exceeds the model limit.

11. Best Practices¶

Industry standards¶

Cosine similarity is the default — pick this unless you have a very specific reason.
Pre-normalize vectors when supported (saves runtime cost).
Version your index with embedder name (e.g., index-openai-3-small-v2).
Cache during dev, persist in production.

Production¶

Use text-embedding-3-small for general production.
Upgrade to -large or specialty models only after measurement justifies it.
For privacy-sensitive: bge-large-en-v1.5 (top open-source, runs locally).
Combine dense + sparse retrieval (Chapter 6) for keyword + semantic hybrid.

Optimization¶

Truncate dims with OpenAI's dimensions param — dimensions=512 gives near-full quality at ⅓ storage.
Batch embedding requests — 100-1000 chunks per API call.
Async embedding in indexing — overlap embedding + insertion.

12. Evolution Story¶

flowchart LR
    A[TF-IDF<br/>keyword frequency] --> B[BM25<br/>better TF-IDF]
    B --> C[Word2Vec<br/>word embeddings]
    C --> D[GloVe / FastText<br/>better word embeddings]
    D --> E[BERT / Sentence-BERT<br/>sentence embeddings]
    E --> F[OpenAI ada<br/>large-model embeddings]
    F --> G[text-embedding-3<br/>state of the art OpenAI]
    G --> H[Voyage / Cohere / Jina<br/>specialized models]
    H --> I[Long-context embedders<br/>Jina v3, late chunking]
    I --> J[Binary embeddings<br/>extreme scale]

Where we are: Modern embeddings make semantic retrieval routine. The cosine similarity floor of 0.7-0.8 reliably means "semantically related" in production systems.

Where we're going (next chapter): Once we have vectors, we need to store them and search at scale. We'll dive into vector databases — FAISS, ChromaDB, Pinecone, Weaviate, Milvus, Qdrant, pgvector — and how HNSW makes million-vector search feel instant.

Practice¶

What does this print?

Expected: 1.0

import numpy as np
a = np.array([0.6, 0.8])
b = np.array([0.6, 0.8])
cos = (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))
print(round(cos, 2))

Compute cosine similarity correctly (divide by BOTH norms)

Expected: True

import numpy as np
a = np.array([3.0, 4.0])
b = np.array([1.0, 0.0])
cos = (a @ b) / np.linalg.norm(a)       # bug: missing norm of b
print(0 < cos < 1)

Quiz — Quick check¶

What you remember

Q1. Cosine similarity of two identical vectors equals…

0
1
-1
Depends on dimensions

Q2. For unit-normalized vectors, dot product equals…

Cosine similarity
Euclidean distance
Manhattan distance
Hamming distance

Q3. Which distance is fast for binary vectors via CPU XOR?

Cosine
Euclidean
Hamming
Jaccard

Common doubts¶

Why not always use the largest embedder?

Cost (2-4× more) + storage (2× the dimensions) + speed (slower indexing and queries). Start small; upgrade only when measurement demands.

Should I average word embeddings to embed a sentence?

No. Modern sentence embedders (text-embedding-3-*, BGE, etc.) handle whole sentences natively. Averaging word embeddings is the 2014 way; quality is dramatically worse.

Can I mix different embedders in one vector store?

No. Different models map to different spaces. Cosine similarity between an OpenAI vector and a Cohere vector is meaningless. One model per store.

→ Vector Stores & Databases