What is a token?

Those three numbers, pricing, context limits, and benchmark cost, all assume the reader already knows what a token is. But even for people who work with these systems daily it would be difficult to define one precisely, and explain why the same sentence produces a different token count on a different model, or say why some languages cost noticeably more to process than others. This post works through the definition, the algorithm that produces tokens, and what that means for context budgets, pricing, and multilingual products.

The definition

A token is the smallest unit of data a language model processes. For English text, one token is roughly four characters or three-quarters of a word.

These are approximations. The exact count for any text depends on the tokenizer the model uses, and identical text produces a different count on a different model.

Tokens can also represent parts of images, audio, or video, covered further down. In the context of language models and API pricing, "token" almost always means a text token, and that's the sense used through the rest of this post.

Why tokens exist

Models process numbers, not text. A tokenizer converts raw text into a sequence of integer IDs the model runs on, and converts the model's output IDs back into text.

The hard part is vocabulary design. A vocabulary with one entry per word would be enormous and would still fail on rare words, technical jargon, and new terminology. A vocabulary with one entry per character stays small but loses semantic coherence and produces long sequences for the same content. Most production models take a middle path: subword tokenization, most commonly Byte Pair Encoding (BPE).

How BPE works

BPE started as a data compression algorithm (Gage, 1994) and was adapted for neural machine translation by Sennrich et al. (2016). It's now the tokenization method behind most major language models.

Many GPT-style tokenizers start from individual bytes (0 to 255) rather than characters. Byte-level BPE handles any text, including emoji, Arabic, Chinese, and code, without an unknown-token fallback. Other tokenizers start from Unicode characters or code points; the starting unit varies by design.

The training process, at a high level:

1. Start with a vocabulary of all individual bytes.
2. Find the pair of adjacent tokens that appears most frequently in the training corpus.
3. Merge that pair into a new token and add it to the vocabulary.
4. Repeat for a fixed number of merges, typically 30,000 to 50,000.

The result: common words and subword patterns become single tokens, and rare words split into known pieces. "Darkness" becomes "dark" and "ness". "Brightness" becomes "bright" and "ness". A model trained on enough text learns to place the "ness" token near other suffix tokens in embedding space, which is where the semantic relationship lives. The integer ID itself carries no meaning; the embedding learned for it does.

What counts as a token

Tokens cover more than word fragments. They also cover punctuation, spaces and line breaks, numbers and symbols, and other special characters. A 34-word sentence can easily run to 40 or more tokens, because spaces, punctuation, and word fragments each consume a token independently of the words around them.

The same word can map to different token IDs depending on where it sits. In most tokenizers, the leading space before a mid-sentence word is part of that token, so the same word gets a different ID at the start of a sentence than it does mid-sentence. A prompt that ends with a trailing space isn't the same input as one without it: the space is already folded into whichever token comes next.

Code tokens

Code tokenizes differently from prose. Indentation, brackets, operator characters, and identifier names each consume tokens independently. A missing bracket or space changes what code does, so token boundaries in code carry semantic weight they don't carry in natural language.

The comparison in that table is the practical point for coding agents. On an average codebase, an agent reading eight source files has spent roughly the same tokens as enola.'s full architecture digest. The eight-file read produces an incomplete, sampled picture. The digest is computed from the parsed dependency graph and doesn't depend on which files happened to get read.

Input, output, and reasoning tokens

A single model request touches three token categories, from input to output.

Input tokens. Everything sent in the request: system prompt, conversation history, tool definitions, file contents, the user message. Processed in parallel as one forward pass through the model.

Output tokens. Everything the model generates: response text, tool call parameters, and any intermediate reasoning steps. Generated sequentially, one token at a time, each depending on the one before it. That sequential generation is why output tokens cost more per token than input, across every major provider.

Reasoning tokens. Introduced by reasoning models: tokens generated internally while the model works through a problem before producing its visible response. Billed as output tokens. A complex reasoning query can require a large multiple of the compute of a standard single-pass response, which matters for cost planning at scale.

Context windows

A context window is the maximum number of tokens a model can hold in its active processing state at once, spanning both input and output. When the window fills, older content gets compacted or dropped.

As context grows, older tokens fall out of scope and the model stops referencing earlier details. That's a structural property of how a fixed-size context window works, not a quality difference between models.

For production systems: agentic workflows accumulate context fast. Each file read, tool result, and prior exchange adds tokens. A 200,000-token window sounds large until an agent starts reading source files and stacking up tool output across a long task.

The language tax

Tokenizer fertility is the research term for average tokens per word, the main metric used to compare how efficiently a tokenizer handles different languages.

Petrov et al. (2023) measured fertility across languages and found that non-Latin-script languages can need 2 to 15 times more tokens per semantic unit than English. Ahia et al. (2023) extended this to commercial APIs and showed that the cost of processing equivalent content varies by an order of magnitude across languages, purely from tokenizer design. The gap is widest for languages with complex scripts or morphology (Arabic, Hebrew, Turkish, Thai) and smaller, but still present, for other Latin-script European languages.

The root cause is vocabulary design. BPE merge operations are driven by frequency in the training corpus, and English dominates most training corpora. Merge rules end up favouring Latin scripts, leaving other languages with higher fertility scores. Research is addressing this directly: parity-aware BPE variants (Foroutan et al., 2025) optimise for the worst-performing language at each merge step, and byte-latent transformer approaches sidestep the static-vocabulary problem entirely. Higher fertility for non-English text remains the production reality for most deployed models today.

Multimodal tokens

The same token-based processing extends past text. Visual models map pixels or voxels from images and video into discrete tokens. Audio models convert clips into spectrograms, processed like images, or into semantic tokens that capture meaning rather than raw acoustic data.

The context-window and cost implications carry over unchanged: more tokens means more compute, more cost, and more context consumed, regardless of the modality behind them.

How to count tokens accurately

For planning: budget one token per four characters of English prose, and one token per eight to twelve characters of typical code.

For exact counts: every major provider exposes a token-counting endpoint or library tied to their specific model's tokenizer. Because tokenizers differ across models and providers, the only count worth trusting is the one produced by the tokenizer that will actually process the request.

Where enola. fits

The coding token table above is the structural argument. An agent searching and reading files to understand a codebase's architecture is doing it expensively and approximately: each file read adds input tokens, each tool invocation adds output tokens, and the agent stops when its approximation feels complete rather than when it actually is complete.

enola. replaces that loop with a single tool call. generate_snapshot builds a deterministic graph from the parsed source. impact_analysis returns the transitive blast radius of a change, computed from dependency edges the extractors resolved deterministically (including framework wiring like DI and routing) rather than inferred from a sample of files. llm_context.md gives the agent a roughly 16,000-token architecture digest that can be cached in the system prompt, so every session starts with an accurate structural picture before the first tool call, at cache-hit pricing instead of standard input pricing (see the token economics of a coding agent for the pricing breakdown).

The token math follows directly: fewer search-and-read rounds, a smaller input footprint, and no correction loop chasing a dependency the agent didn't find on the first pass.

curl -fsSL https://raw.githubusercontent.com/enola-labs/enola/main/install.sh | sh

FAQ

What is a token in AI?

A token is the smallest unit of data a language model processes: for English text, about four characters or three-quarters of a word. A tokenizer turns raw text into integer IDs before the model runs and turns the model's output IDs back into text. Because each model ships its own tokenizer, the same input text produces a different token count on a different model.

How many tokens is a word?

Most common English words are a single token; longer or less common words split into two or more pieces. Words in other languages generally cost more tokens per word than English, because BPE vocabularies are built from training data where English text dominates the merge statistics.

What is byte pair encoding?

BPE is the tokenization algorithm behind most production language models: a 1994 compression technique adapted for translation in 2016. It starts from bytes or characters and repeatedly merges the most frequent adjacent pair into a new vocabulary entry, typically for 30,000 to 50,000 merges, so that any input text, in any script, can be represented without an unknown-token fallback.

What are reasoning tokens?

They're the tokens a reasoning model generates while working through a problem internally, before the response a user sees. Providers bill them as output tokens, and a hard reasoning query can generate far more of them than the visible response length would suggest, which is why reasoning workloads have a different cost profile than standard completions.

What is tokenizer fertility and the language tax?

Fertility is average tokens per word, the metric researchers use to score a tokenizer's efficiency on a given language. The disparity across languages, sometimes called the language tax, runs 2 to 15x for non-Latin scripts versus English per Petrov et al. (2023), and translates into API cost differences of up to an order of magnitude per Ahia et al. (2023).

Can tokens represent things other than text?

Yes. Multimodal models tokenize image patches, audio spectrogram segments or semantic units, and video frame sequences, using the same context-window and per-token cost mechanics as text.

Sources: OpenAI tokenization documentation; NVIDIA Blog, "What Are AI Tokens?" (Dave Salvator, March 2025); Microsoft Copilot, "What are AI tokens?" (March 2025); Sennrich, Haddow, Birch (2016), ACL; Gage (1994), C Users Journal; Hugging Face BPE documentation; Petrov et al. (2023); Ahia et al. (2023); Foroutan et al. (2025).