Meaning, reduced to numbers. Then compared by angle.
You can't feed a word into a neural network. It only takes numbers. Embeddings solve that: they convert words (or sentences, or images, or audio) into vectors of numbers. Dense vectors. Typically 768 to 4096 dimensions, though simpler models use fewer.
The key insight is this: if two vectors are close together in high-dimensional space, their underlying meanings are related. "Dog" and "poodle" are neighbors. "Dog" and "car" are far apart. This is called semantic similarity, and it's the foundation of how LLMs understand meaning.
An embedding is a point in N-dimensional space. The whole game is positioning related concepts near each other so that simple distance metrics (cosine similarity) can tell you if two pieces of text are semantically related.
Early approaches used one-hot encoding: a vocabulary of size N, each word gets a vector of length N with a single 1 and rest 0. Sparse, inefficient, no implicit structure. "Cat" and "dog" would be equally distant from "car" despite sharing conceptual territory.
Word embeddings (Word2Vec, GloVe) improved this: learn dense vectors where similar words cluster. The famous example: king - man + woman = queen. The math works because the vector arithmetic captures analogical relationships.
Modern LLMs generate contextual embeddings: the same word gets different vectors depending on surrounding context. "Bank" in "river bank" vs "investment bank" ends up with different embeddings. This is handled automatically by transformer attention.
You run text through an embedding model. Most commonly: a frozen pretrained model (OpenAI's ada-2, Cohere, open-source options like E5, all-MiniLM). The model reads tokenized text and outputs a vector per token (or averaged/CLS token for sentence-level embedding).
You compare embeddings with cosine similarity, not Euclidean distance. Cosine measures the angle between vectors, ignoring magnitude. This matters because embedding models often normalize outputs to unit length—making cosine equivalent to dot product and computationally cheap.
The list is long:
Semantic search: Embed your documents. Embed the query. Find the nearest document vectors. This replaces keyword BM25 in RAG pipelines.
Clustering: Group related content by embedding proximity. Useful for topic modeling, anomaly detection, deduplication.
Recommendation: "Users who embedded content like X also liked content like Y." Nearest-neighbor search in embedding space.
Classification: Feed embedding vectors into a lighter classifier (or just use nearest-neighbor with labeled examples — zero-shot classification).
Retrieval: The "R" in RAG. You retrieve relevant context by embedding similarity before generation.
Embedding quality depends on the model. A model trained on general web text will embed "financial report" differently than one trained on financial documents. Domain mismatch is a real problem. Fine-tuning embeddings on domain-specific data is a known technique that significantly improves retrieval quality.
Vector dimensions affect quality and speed. Higher dimensions = more expressive = slower nearest-neighbor search. 768-1536 is common for production. 4096 used in frontier models but rarely needed.
Embeddings convert any token sequence into a dense numerical vector. Related concepts cluster nearby in high-dimensional space. Cosine similarity finds semantically related content. This is how LLMs "understand" that two sentences are about the same thing — not through meaning, but through geometry.