Everybody who hears about Chain-of-Symbol prompting files it under "shorter prompts". Fewer tokens, cheaper calls, a micro-optimisation for after the real work. That reading is wrong in a way that makes people implement half the technique and wonder why nothing improved.
The token saving is the second benefit. The first is that a symbolic state has exactly one reading.
Here is the sentence that starts the whole thing, from Hu et al., 2023:
the hall, which lies west of the pantry, has a door on its north side leading to the study
To use that, a reader has to resolve which, resolve its, and carry a coordinate frame in working memory. Three resolutions, three places to be wrong. Chain-of-thought does this on every step of a spatial task, because CoT insists on writing the intermediate state out in prose — and that state is not a calculation, it is a graph. Every restatement re-renders the graph into sentences, and every render is another chance to drop an edge, flip a direction, or attach a pronoun to the wrong room.
The failure is not that the model cannot reason. It is that it is reasoning over a representation that keeps corrupting itself.
I built it as a live page — random world generator, two serializers off one state object, two real parsers, a BFS planner for ground truth, a hand-written tokenizer — so the claim could be measured rather than asserted.
The state object comes first, and the prompt is a view of it
The most common mistake is writing the prompt by hand and letting the world exist only inside that string. You cannot then check it, diff it, or re-serialize it.
// STEP 1 - the state. Everything else is a function of this.
const world = {
rooms: [ {id:0, name:"kitchen", x:0, y:0},
{id:1, name:"hall", x:1, y:0},
{id:2, name:"pantry", x:1, y:1} ],
doors: [ [0,1], [1,2] ], // sorted, [min,max]
objects: [ {name:"key", room:2} ],
start: 0,
goal: "key"
};
Both the prose version and the symbolic version are renderings of that object. They can never drift, and you get a ground truth for free — which is the only reason two prompts can be shown side by side and honestly claimed to encode the same information.
The symbol table is the whole contract
A symbol the model has to infer is worse than the word it replaced, because now it is guessing and you deleted the redundancy that would have let it recover. So CoS opens with a tiny legend, stated once, before any data.
# STEP 2 - the symbol table. This is the whole contract.
# a(x,y)=room at grid cell a/b=door between a and b
# @a=you are here a:o=object o is in room a
# *o=the object to find
# Rules for choosing symbols:
# - one concept per symbol, no overloading
# - ASCII you would find in code: ( ) , / : @ * - >
# - never use a symbol that also appears inside your data
Two properties make this work. It costs a fixed handful of tokens however big the world gets, which is why the saving grows with world size. And it hands the model a notation to copy into its own answer, which is what makes the answer machine-parseable.
Here is what the serializer emits for the page's default world — eight rooms, seed 611, letters scheme:
Answer in the same symbols. Move only along / edges.
# a(x,y)=room at cell a/b=door @a=start a:o=object in a *o=goal
K=kitchen H=hall P=pantry C=cellar S=study A=attic G=garage F=foyer
K(1,1) H(1,2) P(0,1) C(2,1) S(0,2) A(1,0) G(0,0) F(2,2)
K/H K/P K/C K/A H/S H/F P/S P/G
@C S:key K:lantern F:ledger *key
Example: @a -> a/b -> @b DONE
Plan:
Everything is sorted. A serializer that emits doors in hash order gives a different prompt for the same world every run, which destroys prompt caching and makes evals non-reproducible.
The chain half is the half people skip
This is where most implementations quietly throw away the benefit: they symbolize the input, then let the model narrate its reasoning in English. But the chain is where the state is updated step by step, so the chain is where ambiguity does the damage. If the model writes "I move to the room north of here, which also touches the pantry", you are resolving which all over again.
One worked demonstration in the answer language is all the instruction you need.
// STEP 4 - one demonstration, in the answer language you want back.
const DEMO =
`Example:
@kitchen -> kitchen/hall -> @hall -> hall/pantry -> @pantry DONE`;
Each arrow is a state transition, each door symbol the edge used. The model is now doing symbol rewriting rather than prose composition — an easier next-token problem, because the format is rigid and the search space at each step is tiny.
Three ways prose loses an edge
Name the failures rather than waving at "ambiguity" — it is the same three every time. The page's prose parser implements exactly the shortcuts a hurried reader takes, each right some of the time:
-
H1 resolve
which/itsto the nearest preceding room noun -
H2 resolve
itto the most recently mentioned room -
H3 read
your leftallocentrically unless a facing was declared in the previous sentence
"The hall, which lies west of the pantry, has a door on its
north side leading to the study." // H1 -> pantry/study WRONG
"From the study, the vault lies to your left."
// H3 -> assumes facing north
"It also opens east onto the cellar."// H2 -> binds to the wrong room
The page's default world fails on H1 twice. The generator produced this pair:
The hall, which lies east of the study, has a door on its east side leading to the foyer.
The pantry, which lies north of the study, has a door on its north side leading to the garage.
The parser attaches both its to study, the nearest preceding noun. It loses foyer>west>hall and garage>south>pantry, and invents foyer>west>study and garage>south>study in their place. Two edges gone, two hallucinated, out of grammar a human writer produces without thinking. Feed that reconstructed world to BFS and you get a plan through doors that do not exist. The symbolic parse of the same world recovers every relation exactly.
Parse the answer strictly, and fail loudly
A symbolic answer is worth having because it is machine-checkable. Do not regex out the room names and shrug at the rest — validate every transition against the world you serialized, so a hallucinated door is caught in your code, not by a user.
// STEP 5 - parse + VALIDATE. The validation is the point.
function parsePlan(txt, w){
const path = [], seen = txt.match(/@([a-z]+)/g) || [];
for (const t of seen) path.push(t.slice(1));
if (!path.length) throw new Error("no @room tokens in answer");
if (path[0] !== w.rooms[w.start].name) throw new Error("wrong start");
for (let i = 0; i + 1 < path.length; i++){
if (!hasDoor(w, path[i], path[i+1]))
throw new Error("hallucinated door " + path[i] + "/" + path[i+1]);
}
return path; // now provably walkable in the real world
}
A parse failure is worth logging: it usually means the symbol table was ambiguous rather than the model bad — an overloaded glyph, or a symbol that also appears inside your data.
That strictness is the underrated payoff. A prose answer has to be understood before it can be validated; a symbolic answer only has to be parsed — which is what buys you evals with no judge model in the loop, because correctness is decidable.
The attention-distance argument, and where the honesty line sits
There is a second, quieter reason the symbolic form helps, and it survives even if the model resolves every pronoun perfectly. A fact is stated once near the top and used much later, when the plan step needs it. The tokens between statement and use are a real quantity, and there are far fewer of them when the world is 40 tokens of symbols than 400 of prose.
The page measures the true token offset of every edge, runs a decay curve over it, and prints its assumptions on screen rather than hiding them: A1 a fact must be recalled when the step using it is emitted; A2 p(recall) = floor + (1 − floor)·e^(−d/L) with floor = 0.75 and L on a slider; A3 a fact the parser misread is wrong regardless of recall; A4 a trial succeeds only if every fact on the shortest path survives A1–A3.
// measured, not estimated: the real token offset of every edge
const d = totalTokens - factPos["3-7"];
const p = floor + (1 - floor) * Math.exp(-d / L); // TOY model, stated
The offsets are real, measured on the actual strings. The curve over them is a toy with two free parameters, there is no model on the page, and nothing about its shape is evidence of anything. Token counts and parse accuracy are measurements; the solve rate is not. Keeping those halves apart is the difference between a demo and a lie.
The tokenizer trap
"Symbolic" feels like it must be shorter, so people reach for beautiful rare glyphs — arrows, mathematical brackets, box-drawing — and quietly make the prompt more expensive than the prose it replaced.
kitchen(0,0) # names + coords - readable, no legend
K(0,0) # letters + legend - usually the cheapest
⟨K⟩⟦0,0⟧ # exotic glyphs - looks symbolic, costs MORE
The page's tokenizer is hand-written and documented rule by rule so it can be re-implemented independently, and the rule that bites is R5: every non-ASCII character costs 2 tokens, because rare codepoints fall back to byte-level pieces. Tokenizers are trained on text, and common ASCII punctuation is cheap.
So K(0,0) tokenizes to ["k","(","0",",","0",")"] — six tokens — while ⟨K⟩⟦0,0⟧ is twelve. Same information, double the bill, and the parse is perfect either way. The encoding changed; nothing else did.
What I actually measured
I pulled the page's core out of the HTML — it is DOM-free and exports itself under node for exactly this reason — and ran it headlessly.
The round-trip assertion first, because it is the non-negotiable one. Serialize, parse, deep-equal the original, over 500 random worlds of 4 to 14 rooms, for all three schemes: 1,500 assertions, zero failures. If that ever fails, your scheme is ambiguous and no prompt tuning fixes it.
Then the prose parser on the same worlds. At verbosity 0, where the generator may only emit "The B is dir of the A." sentences, it recovers the world exactly 500 times out of 500. At verbosity 3, with relative clauses, egocentric directions and long-range pronouns switched on, it manages 31 out of 500. That gap is the whole argument: the parser did not get worse, the encoding did. Symbols read 100% at every verbosity, because there is no antecedent to find and no frame to carry.
Across the default benchmark — 9 world sizes, 8 worlds each, 72 worlds, verbosity 2, letters scheme — the average prose prompt is 225 tokens against 183 symbolic, an 18.8% cut, while exact world recovery is 26.4% for prose and 100% for symbols. At verbosity 3 the cut widens to 29.3% and prose recovery collapses to 8.3%. The saving grows with world size — 9% at four rooms, 24% at twelve — because the symbol table's cost is fixed while prose keeps paying per sentence.
And the trap, measured: the same 72 worlds under the exotic-glyph scheme come out at 394 tokens against 225 for prose — 75% more expensive — with parse accuracy still a flawless 100%. Pretty glyphs bought nothing and cost double.
One honest divergence. The page calls the letters-plus-legend scheme "usually the cheapest", but under its own tokenizer it never beats plain names at any size from 4 to 16 rooms — 130 against 125 at four rooms, 292 against 279 at sixteen. The legend line never amortises, because this tokenizer charges a short word like kitchen only two pieces. Which is exactly the lesson: measure with the real tokenizer for your model, and trust no note about it, including that one.
What the paper measured
None of the above is the evidence. The evidence belongs to somebody else.
Hu, Chen, Wu, Ru, Wan, Sun, Cheng, Wong — "Chain-of-Symbol Prompting Elicits Planning in Large Language Models" (2023) — build SPP, a spatial-planning benchmark of three simulated environments: Brick World, NLVR-based Manipulation, and Natural Language Navigation. They compare chain-of-thought written in natural language against the same chains written in condensed symbols. On Brick World with ChatGPT they report accuracy rising from 31.8% to 92.6%, with prompt tokens falling by up to 65.8%.
They are careful that the gain is not uniform: it depends on the number of demonstrations and on the model. CoS is a representation change, not a reasoning upgrade.
When symbols are the wrong answer
CoS strips out the thing that makes an LLM useful on a lot of tasks: the meaning carried by the words. If the question depends on a kitchen having a fridge, or "fragile" changing how a box is handled, replacing that noun with K deletes the prior you were paying for.
USE CoS when:
- the state is combinatorial and gets restated every turn
- relations are the payload (adjacency, order, containment, coords)
- you can check the answer in code afterwards
- prompts are long enough that attention distance is real
DO NOT use CoS when:
- the answer depends on what the words MEAN, not how they relate
- the state is small enough to state plainly in three sentences
- your users read the prompt (symbols are hostile to humans)
Symbolize the structure, keep the words that carry semantics. pantry(1,1) pantry:fragile-crate is not a compromise — it is an accurate description of which half of your problem is relational and which half is meaning.
The uncomfortable question at the bottom
Once your world is a proper graph in your code, BFS gives you the provably optimal plan in ten lines. So why ask a model at all? The honest answer is that CoS is most valuable where the state extraction is the hard part and the search is easy.
# the architecture CoS actually unlocks:
symbols = llm_extract(messy_description) # model does the reading
world = parse_strict(symbols) # your parser validates
plan = bfs(world) # your code does the search
Read a messy human description into your symbols with the model, verify the extraction, then plan in code. That is far more robust than asking one model to both understand and search, and the symbolic interface is what makes the handoff checkable. The seam is the symbol table.
Slide the verbosity to zero, watch the prose parser recover to 100%, then slide it back up and watch a single relative clause corrupt an edge: https://dev48v.infy.uk/prompt/day61-chain-of-symbol.html
