Embeddings: Turning Meaning into Vectors

Introduction

Every semantic search engine, every RAG pipeline, every recommendation system built in the last decade rests on one idea: represent meaning as geometry. Convert text into a vector of numbers such that texts with similar meanings land close together in a high-dimensional space — then “find similar things” becomes “find nearby points.”

That conversion is an embedding. This mission builds the mental model from scratch, because everything in Levels 5–8 of this roadmap — vector databases, hybrid search, reranking, retrieval evaluation — assumes you think in these terms fluently.

Learning objectives

• Explain what an embedding is and what the dimensions represent.
• Use cosine similarity to compare meanings, and know why it beats Euclidean distance.
• Understand how embedding models are trained (contrastive learning) — enough to predict their failure modes.
• Choose an embedding model deliberately, knowing the re-embedding cost of switching.

From text to coordinates

An embedding model is a neural network with a peculiar job: swallow text, emit a fixed- length vector — typically 384 to 3,072 numbers. The input can be a word, a sentence, a paragraph, or a whole document; the output is always the same shape.

from openai import OpenAI

client = OpenAI()
resp = client.embeddings.create(
    model="text-embedding-3-small",
    input="The reactor must be calibrated before each batch.",
)
vector = resp.data[0].embedding   # list[float], len == 1536

What do the 1,536 numbers mean? Nothing individually — and everything collectively. Each dimension is a learned feature the network found useful for distinguishing meanings. Think of them as coordinates: the model has placed every piece of text it has ever seen somewhere in a 1,536-dimensional space, arranged so that distance ≈ semantic dissimilarity.

The classic intuition: imagine a 2-D map where “king” and “queen” are neighbors, “king” and “sandwich” are not. Real embedding spaces do this in thousands of dimensions, and the “map” encodes much richer relationships than any 2-D analogy — but the geometry-is-meaning idea is exactly right.

Similarity: cosine, not distance

To compare two embeddings you almost always use cosine similarity — the cosine of the angle between vectors, from −1 (opposite) through 0 (unrelated) to 1 (same meaning):

$$ \cos(\theta) = \frac{A \cdot B}{|A| , |B|} $$

Why cosine instead of straight-line (Euclidean) distance? Because cosine measures direction, not magnitude — and in text, magnitude often encodes artifacts (text length, frequency effects) rather than meaning. Two documents about calibration should point the same direction whether one is a sentence and the other is a manual.

Practical note: most modern embedding models emit unit-normalized vectors (length 1), which makes cosine similarity a plain dot product — and makes vector indexes fast.

import numpy as np

def cosine(a: list[float], b: list[float]) -> float:
    a, b = np.asarray(a), np.asarray(b)
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

q  = embed("how do I calibrate the reactor?")
d1 = embed("Calibration procedure for reactors")     # ~0.62 — relevant
d2 = embed("Quarterly financial report")             # ~0.11 — unrelated

How embedding models learn

Embedding models are trained with contrastive learning: show the network pairs of texts that mean the same thing (pull their vectors together) and pairs that do not (push them apart), over billions of pairs. The result is a model that has internalized paraphrase, topic, and entailment — without anyone labeling dimensions.

This training story predicts the failure modes you will hit in production:

• Similar ≠ relevant. Cosine similarity measures surface relatedness. A query about “Python” retrieves snake-care docs and programming docs with similar scores. Meaning overlap is not answer relevance — which is why Level 5 adds reranking and filtering.
• Domain drift. A general-purpose model may place specialized jargon (pharma batch records, MES error codes) in unhelpful positions. Domain-adapted models exist for this.
• No exact matching. Embeddings are fuzzy by design. Error codes, part numbers, and IDs do not embed meaningfully — the root cause behind hybrid (vector + keyword) search.

Choosing a model — deliberately

Your choice of embedding model is an architecture decision, not an implementation detail, because of one brutal fact: the index only makes sense with the model that created it. Change the model and every vector in your database is garbage — you must re-embed the entire corpus.

Consideration	What to weigh
Dimensionality	Higher dims: more nuance, more storage, slower search. 384–768 is plenty for most corpora.
Leaderboards	MTEB ranks models on retrieval benchmarks — a shortlist, not a decision.
Domain fit	Code, medical, or multilingual corpora may need specialized models.
Cost & hosting	API models (OpenAI, Cohere, Bedrock) vs. self-hosted (sentence-transformers, BGE, E5).
Privacy	Self-hosted models keep text inside your network — often decisive in regulated domains.

The engineering discipline: treat the embedding model as a versioned dependency. Record which model built each index; store the model name alongside vectors as metadata; plan re-embedding as a routine ETL job, not an emergency.

Where embeddings show up

• Semantic search / RAG — the primary use; Level 5 builds the full pipeline.
• Clustering and deduplication — near-duplicate detection via vector proximity.
• Classification — embed once, train a tiny classifier on the vectors (cheap, robust).
• Anomaly detection — outliers live far from every cluster.
• Recommendations — “more like this” as a nearest-neighbor query.

One mental model, many products — which is why this is a Level 3 mission and not a Level 5 footnote.

Hands-on exercise

Build a minimal semantic search over your own notes — no vector database yet, just NumPy (that comes in Level 5):

1. Collect 20–50 text snippets (notes, docs, READMEs) in a list.
2. Embed them all with any API or sentence-transformers model; store vectors in a 2-D NumPy array, unit-normalized.
3. Embed a query; compute cosine similarity against every row; return the top 3.
4. Test the boundary: find a query where the top hit is related but wrong. Write down why the embedding space confused them.
5. Swap the embedding model and repeat. Observe how rankings change — that instability is why model choice is an architecture decision.

Twenty lines of code, and you have built the core loop of every vector database.

Key takeaways

• An embedding maps text to a fixed-length vector where distance encodes meaning.
• Use cosine similarity — direction, not magnitude — for comparing embeddings.
• Models learn via contrastive training; their failure modes (similar-but-irrelevant, domain drift, no exact match) follow directly from that training.
• Choosing an embedding model is an architecture decision: switching means re-embedding everything.
• The nearest-neighbor loop you built here is the engine inside pgvector, Milvus, and every vector database you will meet in Level 5.

Continue your journey

Next mission: Attention Mechanisms and Transformers (coming soon) — or continue to Quantization: FP32, FP16, INT8, and 4-Bit Models. Full journey: roadmap overview.