Tokens: The Currency of AI

When you ask a chatbot for a recipe, it does not see words. It sees tokens. Every phrase you type gets chopped into small, numbered pieces. Those pieces are the thing you pay for, the thing that fills the model’s working memory, and the reason your assistant sometimes forgets what you said three paragraphs ago.

A single English word can break into half a dozen tokens in one model and remain whole in another. That mismatch silently inflates your API (application programming interface) bill, truncates your prompt mid-sentence, and explains why your chatbot “forgets” earlier instructions. This article shows you exactly what a token is, where it comes from, and why treating tokens as a first-class resource changes how you design AI systems.

What exactly is a token?

A token is the smallest unit an LLM (large language model) reads. You hand it text. It maps that text to a list of integers. Each integer points to an entry in a fixed vocabulary of subwords, characters, or control symbols.

Think of tokens as the currency of AI. You do not pay for a conversation in words or paragraphs. You pay in tokens. The model processes tokens. It gets confused when tokens run out. Embedding tables, attention layers, and the final output all operate on token indices, not on the characters you typed. When that fact is invisible, you misjudge cost, truncation, and what a model can actually remember.

The vocabulary is built once, at training time, from a huge corpus. It might contain 50,000 to 250,000 entries, each learned to cover frequent strings compactly and rare strings by composition. A token can be a full word like “the”, a morpheme like “ing”, a punctuation mark, or a special marker that tells the model where the system prompt ends. The model never sees raw text. It only ever sees sequences of these integer IDs. tiktoken

Why does a chatbot split "strawberry" into three pieces?

A chatbot splits “strawberry” into multiple tokens because of something called subword tokenization. Instead of keeping every possible word in the vocabulary, the tokenizer learns reusable chunks. Common words stay whole. Rare words get built from smaller parts. That way the model can read any string you throw at it, even typos or made-up words, without ever running into an “unknown” token.

The dominant algorithm is Byte Pair Encoding, or BPE. BPE starts from raw bytes or characters. It scans a massive amount of text and repeatedly merges the most frequent adjacent pair into a new token. After thousands of merges, the vocabulary contains short, frequent sequences like “st”, “raw”, and “berry”. When you feed in “strawberry”, the greedy BPE decoder applies merges in the order they were learned. It finds that “st” matches the earliest applicable merge, then “raw”, then “berry”. So the word becomes three tokens. BPE paper

Here is what that looks like with OpenAI’s tiktoken library for GPT-4:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4")
tokens = enc.encode("strawberry")
# tokens is something like [72, 75, 76]
for t in tokens:
    print(enc.decode_single_token_bytes(t))
# b'st'  b'raw'  b'berry'

The same logic applies to your recipe. “Give me a recipe for chocolate chip cookies” might tokenize as “Give”, “ me”, “ a”, “ recipe”, “ for”, “ chocolate”, “ chip”, “ cookies”. Each space in the output shows that the tokenizer includes leading spaces as part of the token. That is a byte-level detail that makes token boundaries unintuitive but guarantees every possible byte sequence is representable. OpenAI tokenizer

How does tokenization convert my recipe request into numbers?

Every interaction with a chatbot follows a deterministic pipeline. Text goes in, integer IDs come out. The IDs then drive everything downstream: embedding lookups, positional encodings, and the model’s own computation.

The pipeline works like this:

flowchart LR
    A[Raw text input] --> B[Normalize & byte-encode]
    B --> C[Pre-tokenize on whitespace/punctuation]
    C --> D[Apply BPE merges greedily]
    D --> E[Map subword strings to token IDs]
    E --> F[Sequence of integers fed to model]

First the tokenizer normalizes the string. It might fold case, strip zero-width characters, or enforce one Unicode normalization. Byte-level BPE, used in GPT-2 and later, maps every Unicode code point to its UTF-8 bytes and treats each of the 256 possible bytes as the base alphabet. That step alone guarantees no unknown characters can break the pipeline. byte-level BPE

Next, a pre-tokenizer splits the text into candidate words using whitespace and punctuation. “Give me a recipe for chocolate chip cookies.” becomes a list of seven pieces. Within each piece, the BPE merges run. “chocolate” might stay whole because it is frequent. “cookies” might fragment into “cook” and “ies”. The merge rules are stored as a large hash table; tokenization is a fast, table-driven lookup that never depends on the full vocabulary ordering at runtime.

Finally, each subword string gets mapped to a unique integer ID by a trie or hash map. That sequence of IDs is what the model consumes. The total number of IDs produced is the token count for your request. You can check it yourself with tiktoken before you ever send a prompt to the cloud. Hugging Face tokenizers

Last week we saw that an LLM predicts the next word. Here you can see what “word” really means: it means the next token in this ID sequence. Everything the model learns about grammar, recipes, and style is anchored to these integer indices, not to the English letters you typed.

Why is pricing per token, not per word?

Pricing is per token because the computer works per token. Every token in your prompt triggers a table lookup in the embedding matrix, a row of the positional encoding, and a cascade of matrix multiplications through attention and feed-forward blocks. When the model generates a response, each new token you receive required a full forward pass. Charging per token aligns billing directly with the compute you consume. OpenAI pricing

A “word” is a fuzzy, language-dependent notion. “Pneumonoultramicroscopicsilicovolcanoconiosis” is one English word but could be six tokens. Token counts do not depend on anyone’s definition of a word; they are machine-verifiable. That makes them a sane billing unit across languages, code, logs, and mixture of all three.

Using the recipe example: suppose your prompt “Give me a recipe for chocolate chip cookies” tokenizes to 11 tokens. With a completion that generates another 200 tokens, your total consumption is 211 tokens. If the model charges $0.01 per 1,000 input tokens and $0.03 per 1,000 output tokens, you can predict the exact cost of that recipe interaction before you write a single line of code.

A subtle corollary: phrases that look short but contain rare subwords can cost more than longer, simpler sentences. That is because rare words break into many subword pieces. Here is a side-by-side comparison:

{
  "type": "bar",
  "title": "Token count for different phrases",
  "caption": "Even short phrases can have varying token counts depending on word rarity.",
  "data": [
    {
      "label": "Hi there.",
      "value": 3
    },
    {
      "label": "Antidisestablishmentarianism",
      "value": 5
    },
    {
      "label": "let me write you a short poem",
      "value": 7
    }
  ]
}

The token count, not the character count, is what empties your wallet.

What is the context window, and why does it matter for my chatbot?

The context window is the maximum number of tokens the model can process in a single forward pass. It counts every token: your system prompt, the message history, the current query, and every token the model generates so far. Once the total exceeds that limit, earlier tokens are simply not visible to attention. The model cannot recall them.

If you use a model with a 128k-token context window, you might think you can dump an entire book in the prompt. But if that book is full of rare words that break into many subword tokens, the actual token count might be far higher than the word count suggests. Even a few thousand “words” can eat a surprising fraction of the budget. When the window fills, your recipe from the top of the conversation vanishes, and the model starts answering as if it never saw it. Gemini context window

This is not a failure of memory in a psychological sense. It is a hard resource constraint built into the transformer architecture. Self-attention scales at least quadratically with token count, so the context window is both a limit on what the model can see and a key driver of latency and cost. The wider the window, the more expensive every request becomes.

Engineers who think in tokens check the token count of every prompt before it hits the model. They truncate old messages strategically, summarise lengthy histories, or split long documents into overlapping chunks that each fit comfortably inside the window. In all cases, the first step is to run the exact tokenizer and count.

How does token-level caching save money and latency?

When you send the same system prompt over and over, the model does the same work each time. The matrices that represent the prompt’s tokens are identical from request to request. Modern inference engines exploit that by caching the key-value tensors produced by attention layers for every prefix. These cached tensors are tied directly to the token sequence. If the first N tokens do not change, the model can skip recomputation and jump straight to processing the new tokens. KV caching in Hugging Face

This caching is not a vague concept of “reusing past answers.” It is a mechanical reuse of large, numerical arrays that map one-to-one with token positions. The cache is indexed by token offset in the sequence. Insert a different token at position 0, and the entire prefix cache becomes invalid. Keep the system prompt word-for-word identical, and you can reuse every single one of those tensors.

In the chatbot example, imagine you build a cooking assistant. Its system prompt is a 200-token block that sets the assistant’s tone and gives kitchen safety rules. Without caching, every new recipe request pays to reprocess those 200 tokens. With prompt caching enabled, you pay only for the new user message and the generated tokens. That often cuts latency in half and slashes the per-request cost dramatically. OpenAI prompt caching

graph TD
    A["System prompt tokens"] --> B["Compute KV cache"]
    B --> C["Store cache"]
    D["User message tokens"] --> E["Append to cached prompt"]
    C --> E
    E --> F["Model forward pass"]
    F --> G["Response tokens"]

The takeaway is: when you control your prompt’s token structure carefully, you unlock deep performance wins. Always keep the static prefix at the beginning. Never insert dynamic content before the cached portion. Treat the token sequence as a precious resource whose alignment with the cache saves real money.

Quick reference

{% raw %}

{
  "type": "stat",
  "title": "Key token metrics",
  "caption": "Approximate ratios and sizes for typical LLMs.",
  "stats": [
    {
      "value": "~0.75",
      "label": "Words per token"
    },
    {
      "value": "~4",
      "label": "Chars per token"
    },
    {
      "value": "50k to 250k",
      "label": "Vocabulary size"
    },
    {
      "value": "Up to 128k",
      "label": "Context window"
    }
  ]
}

Test yourself

You are building a summarisation tool that accepts user-pasted articles. A user submits a 2,500-word news story. You plan to process it with GPT-4o (128k context) and generate a 150-word summary. What token-related concerns should you address before writing the API call? How would you verify your assumptions?

Answer: The 2,500-word input likely expands to roughly 3,300 tokens (assuming ~0.75 tokens per word) plus overhead for a system prompt and instruction, say another 100 tokens. The total prompt tokens are well under the 128k limit, so no truncation is expected. However, the model’s output of ~200 tokens will be charged at a higher rate. You should verify the actual token count using tiktoken before deployment, because domain-specific vocabulary (legal terms, names) might splinter into more subwords and inflate the count. Also check that your integration truncates old conversation turns if the same session later reuses the summary; if the user uploads multiple articles, the accumulated token load could silently push earlier input out of the window.

Frequently Asked Questions

Q: Why does my sentence sometimes cost more tokens than I expected?
Rare words, technical jargon, or non-English text often break into multiple subword tokens. Two sentences of equal character length can differ by 30% or more in token count. Always measure with the actual tokenizer, never estimate from character or word counts.

Q: Do all models use the same tokenizer?
No. Each model family typically trains its own tokenizer on its own data. GPT-4 uses a byte-level BPE with roughly 100k tokens. LLaMA-3 uses a SentencePiece tokenizer with a vocabulary around 128k. Even two BPE tokenizers will segment the same text differently if they were trained on different corpora or with different merge thresholds.

Q: Can I count tokens myself before sending a request?
Yes. Libraries like tiktoken (for OpenAI), Hugging Face tokenizers, and SentencePiece expose the exact same encoding as the model. Call .encode() and check the length. Many API providers also return the token usage in the response, but pre-counting avoids sending over-budget requests in the first place.

Q: What happens if I exceed the context window?
The request will be either rejected by the API or silently truncated. Truncation often keeps the latest tokens and drops the oldest, meaning your earliest instructions vanish. Some providers warn you; others do not. Always design prompts so the total token count never approaches the limit.

Q: Is there any advantage to a smaller vocabulary and longer sequences?
A smaller vocabulary makes the embedding matrix cheaper and guarantees full byte-level coverage without a large number of rare tokens. But it forces the model to process more tokens per message, increasing the quadratic self-attention cost. The right size is an empirical trade-off that different model teams settle on during training.

Keep learning how real systems work

If you want a breakdown like this every week, the engineering behind the APIs you call, subscribe to Internals Decoded. We publish one deep article each week, always sourced from primary code and spec. No recaps. No fluff. Just the internals that make the difference in production.

Sources

• tiktoken GitHub
• BPE: Original paper on neural machine translation of rare words with subword units
• OpenAI tokenizer playground
• Byte-level BPE (GPT-2 paper)
• Hugging Face tokenizers library
• OpenAI pricing
• Gemini long-context models
• Past key values in Hugging Face generation
• OpenAI prompt caching announcement

Originally published at Internals Decoded. AI internals, explained conversationally.