Text goes in. Numbers come out. Nothing is magic.
You write "The quick brown fox." The AI sees numbers. Specifically, a sequence of integers—each representing a token. A token is a chunk of text. It could be a word. It could be part of a word. It could be punctuation. The model doesn't care. It only sees numbers.
Tokenization is the process that converts your text into those numbers. Every AI system you've ever used does this. The specific scheme matters enormously.
Tokenization is the bridge between human-readable text and machine-readable numbers. The quality of that bridge affects everything: cost, speed, capability, and the model's ability to handle edge cases like misspellings or rare words.
Simple tokenization—splitting on spaces and calling each a token—seems logical. But it has problems:
Vocabulary size: English has hundreds of thousands of words. A word-based vocabulary would be enormous.
Rare words: Most words appear rarely. A model can't learn meaningful patterns from words it barely sees.
Morphology: "run", "runs", "running", "ran" are four different tokens. A model has to learn separately that they all share a root.
Subword efficiency: Breaking "unbelievable" into "un ##believ ##able" lets one model handle derivation productively instead of storing every variation.
OpenAI's original GPT models used BPE (with a GPT-2 variant called BPE-PSR). BPE starts with individual bytes and iteratively merges the most frequent adjacent pair into a new token. This produces a vocabulary of ~50K tokens, optimized for training data frequency.
Used by BERT and other Google models. Similar to BPE but builds merges based on maximizing training data likelihood rather than pure frequency. Tokens are marked with "##" if they appear as a suffix/middle of a word. Vocab sizes typically 30K-50K.
Used by modern frontier models (GPT-4o, Llama, etc.). Operates at the byte level rather than character level. This means the vocabulary covers all possible byte sequences—256 base tokens—then merges the most useful ones. Benefits:
Any UTF-8 string can be encoded. Even emoji. Even multilingual text. A byte-level vocabulary is complete by definition.
CJK languages (Japanese, Chinese, Korean) use characters that BPE struggles with. Byte-level encoding handles them uniformly.
Tokenization directly affects your API bill. Most pricing is per 1K tokens. Average English text runs about 1 token per word. But it varies:
Prompt engineering tip: token-efficient phrasing costs less to run. "Explain X" vs "Could you please explain what X is and how it works" — the latter is more tokens, same capability delivered.
Tokenization breaks in interesting ways:
These edge cases are why prompt injection and adversarial prompts sometimes work — small token-level changes can bypass safety training in non-obvious ways.
Tokenization converts text to integers. The scheme (BPE, WordPiece, BBPE) determines vocabulary size, subword granularity, and how the model handles unknown input. Your cost per query is literally a function of how many tokens your prompt and response contain. Tokenization isn't glamorous. But it's the first thing that happens to your text.