What Actually Gets Tokenized in SFT

tok.apply_chat_template(msgs, add_generation_prompt=True)

I have copied that line more times than I can count without thinking about what any of it does. The official example writes it that way, the output looks right, you move on.

Then you have to build your own SFT data and compute your own loss mask, and it turns out every piece of that line is load-bearing. I spent today pulling it apart. Notes below.

The counterintuitive part first: the model has no idea messages exists.

TL;DR

• The model only ever sees the rendered string. messages is for Python. Roles are real tokens in the vocabulary.
• A chat template is a Jinja2 program, stored in tokenizer_config.json or as a standalone chat_template.jinja. When both exist the file wins.
• Training renders the full conversation with add_generation_prompt=False. Inference renders msgs[:-1] with add_generation_prompt=True.
• The prefixes those two produce must be identical down to the byte. One assert catches it.
• Don't let the tokenizer add special tokens after templating, or you get two BOS.

The model has no idea messages exists

I had some vague picture in my head where the role was structured metadata and something inside the model read it. Nope.

[{"role": "user", "content": "What is the Young's modulus of graphene?"}]

That list lives in your Python process and nowhere else. What reaches the model is:

[gMASK]<sop><|user|>
What is the Young's modulus of graphene?<|assistant|>

and after tokenization:

[151331, 151333, 151336, 198, ...content ids..., 151337, 198]

Those leading numbers are GLM-4's ids for [gMASK], <sop>, <|user|> and <|assistant|>. Every model family has its own, so don't hardcode them across models.

The role is a token, not a field. The model knows it is its turn to talk because it sees id 151337, the same way it learns any other pattern. No JSON at inference time, no key lookup, no schema. One sequence.

Nearly everything below follows from that.

Jinja2, the rendering layer

Jinja2 is a text templating engine originally built to render HTML for Flask. Three constructs, that's it:

HuggingFace borrowed it for one job: flatten structured messages into the string the model was trained on, and flatten it the same way every time. The minimal shape:

{%- for message in messages %}
    {{- '<|' + message['role'] + '|>\n' + message['content'] + '<|endoftext|>\n' }}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|assistant|>\n' }}
{%- endif %}

The dashes aren't style

{%- and -%} strip whitespace around the tag. Real templates are covered in them.

I assumed this was a formatting preference. It isn't. Templates get indented so humans can read them, and without stripping, every newline and indent in the source lands verbatim in the prompt. One extra \n and the token sequence no longer matches what training saw.

If you write your own, print the result with repr() rather than trusting your eyes.

What's in scope inside a template

transformers renders in a jinja2.sandbox.ImmutableSandboxedEnvironment, so arbitrary attribute access and side effects are blocked. You can't call random Python from in there. What you get:

• messages, plus tools when tool definitions are passed
• add_generation_prompt
• bos_token, eos_token and friends, from the tokenizer
• raise_exception(), for templates that reject malformed conversations, like a system message in the wrong slot
• strftime_now(), for templates that inject the current date

Where the template lives

Two places, same string.

The old way is inline in tokenizer_config.json. That's the file AutoTokenizer.from_pretrained() reads, and note it does not hold the vocabulary. That's in tokenizer.json, or vocab.json plus merges.txt, or a SentencePiece *.model. It only records how to construct the tokenizer object:

{
  "tokenizer_class": "PreTrainedTokenizerFast",
  "model_max_length": 131072,
  "padding_side": "left",
  "bos_token": "[gMASK]",
  "eos_token": "<|endoftext|>",
  "pad_token": "<|endoftext|>",
  "clean_up_tokenization_spaces": false,
  "added_tokens_decoder": {
    "151329": { "content": "<|endoftext|>", "special": true, "normalized": false },
    "151336": { "content": "<|user|>",      "special": true, "normalized": false }
  },
  "chat_template": "{%- for message in messages %}..."   // ← inline, one giant line
}

For SFT work, a few fields are worth a look.

added_tokens_decoder decides whether <|user|> is one atomic token or gets shredded into BPE pieces. Shredded, the role marker stops being a clean signal and the model has to infer the boundary from "< then | then user". It can learn that. No reason to spend the capacity.

padding_side is right for training and must be left for batched generation. In a right-padded generation batch every sequence shorter than the longest one comes out garbage.

eos_token and pad_token govern both where generation stops and which positions get carved out of the loss.

The new way is a standalone chat_template.jinja, which newer transformers versions write by default from save_pretrained(). The reason is mundane: inside JSON every newline is \n and every quote is escaped, so a 200-line template becomes one unreadable line with no diff and no highlighting.

The file wins on precedence. Plenty of repos ship both for backward compatibility, which leaves a trap. Once the two drift apart, behavior depends on your transformers version, and that is a miserable thing to track down. Check they agree before you debug anything else.

add_generation_prompt: training vs inference

This is where I actually got stuck today. One SFT sample:

msgs = [
    {"role": "user",      "content": "What is the Young's modulus of graphene?"},
    {"role": "assistant", "content": "About 1 TPa."},
]

At training time you have the answer. That's the supervision signal, so you render the whole thing with no generation prompt:

tok.apply_chat_template(msgs, add_generation_prompt=False)

<|user|>
What is the Young's modulus of graphene?<|assistant|>
About 1 TPa.<|endoftext|>

Setting it to True would append a second, empty <|assistant|>\n after the answer. A dangling role marker, which the model would learn to emit.

At inference you don't have the answer. Producing it is the point, so you pass msgs[:-1]:

tok.apply_chat_template(msgs[:-1], add_generation_prompt=True)

<|user|>
What is the Young's modulus of graphene?<|assistant|>

add_generation_prompt=True is what appends that trailing <|assistant|>\n. Without it the last token the model sees is ?, and it will cheerfully continue the user's turn, inventing a follow-up question instead of answering. With it, the model is standing exactly where, during training, the next token was the start of an answer.

The prefixes have to match exactly

Line the two up:

training:   <|user|>\nWhat is ... graphene?<|assistant|>\n | About 1 TPa.<|endoftext|>
inference:  <|user|>\nWhat is ... graphene?<|assistant|>\n | ← generation starts here
            └──────────── must match exactly ───────────┘

Everything left of the bar is the model's conditioning context. Training taught it "given this prefix, emit About". If inference rebuilds that prefix with one extra newline, or a space the training path didn't have, the model is conditioning on something outside its training distribution.

What makes it nasty is that it degrades invisibly. Print both strings and they look identical. Nothing in the logs.

So just assert:

full   = tok.apply_chat_template(msgs, tokenize=True, add_generation_prompt=False)
prefix = tok.apply_chat_template(msgs[:-1], tokenize=True, add_generation_prompt=True)
assert full[:len(prefix)] == prefix

Three lines, and it hands you the loss-mask boundary for free: len(prefix) is where the answer starts.

Cutting the loss mask

Roles are tokens, not fields, so there are no field boundaries to slice on. You locate the assistant span yourself.

The direct approach is the length difference:

labels = [-100] * len(prefix) + full[len(prefix):]

The boundary lands naturally after <|assistant|>\n. Clean.

The cleverer-looking alternative is to regex the rendered text for <|assistant|> and map character offsets back to token indices. Don't. The moment a message's content legitimately contains that string it's wrong, and wrong silently.

The length-diff trick has one real limit: it only handles the last turn. For multi-turn data where you want loss on every assistant reply, you'd render incrementally turn by turn, which gets ugly fast.

The proper fix is to mark the assistant span in the template itself:

{%- if message['role'] == 'assistant' %}
    {% generation %}{{- message['content'] + '<|endoftext|>' }}{% endgeneration %}
{%- endif %}

Then ask for the mask:

out = tok.apply_chat_template(
    msgs, tokenize=True, return_dict=True,
    return_assistant_tokens_mask=True,
)
mask = out["assistant_masks"]   # 1 on assistant tokens

Correct for any number of turns. The cost is that the template has to contain {% generation %} blocks, and a lot of published templates don't, which is why I haven't switched to it yet. Check first, or add the blocks to your own copy.

Things that bite

Double BOS. The template already emitted BOS. Call tok(text) after it and the default add_special_tokens=True adds another:

text = tok.apply_chat_template(msgs, tokenize=False)
ids  = tok(text, add_special_tokens=False)["input_ids"]   # ← required

Or skip the round trip with tokenize=True, which uses add_special_tokens=False internally anyway.

Hand-built strings. Writing f"<|user|>\n{q}<|assistant|>\n" in your data pipeline works, right up until the official template changes, or you swap base models, or someone adds a system prompt. Let the template be the single source of truth and stop thinking about it.

Inconsistent template branches. Some hand-edited templates emit <|assistant|>\n inside the loop and <|assistant|> in the add_generation_prompt branch, one newline short. The official GLM and Qwen templates are fine, but if you've touched a template, or added tool definitions or a system prompt, that assert is the only thing that will tell you.

Special tokens in user content. Jinja does plain string concatenation with no escaping. A content field containing the literal text <|assistant|> tokenizes into the real special token, and the model reads a genuine turn boundary. Prompt injection, training-data edition. If your corpus is scraped or model-generated, scan it first.

Wrapping up

It comes down to one thing: the byte string you train on and the byte string you infer on have to share an identical prefix. Jinja2 produces that string, tokenizer_config.json and chat_template.jinja hold the recipe, and add_generation_prompt is the only difference the two paths should have.

Honestly, none of this comes up most of the time. Use a stock model on a standard pipeline and apply_chat_template handles everything; you never need to know what's underneath. It only surfaces when you start building your own data, computing your own masks, or swapping base models. And when it does surface it doesn't throw. It just makes things slightly worse, which is exactly why it's worth knowing in advance.

There's an isomorphic problem on the serving side. The QPS-as-load-signal mistake in Little's Law and vLLM autoscaling has the same shape: a metric that looks reasonable and is quietly measuring something else.

Further reading: the chat templating guide in the transformers docs, and the whitespace-control section of the Jinja2 template designer reference.

中文版：Chat Template 到底 tokenize 了什么