Chain-of-thought: making a model think before it answers

Add four words — "let's think step by step" — to a prompt and a model that was getting a reasoning problem wrong starts getting it right. That this works at all tells you something deep about how LLMs reason, and when they can't.

Chain-of-thought is one of the most powerful and most misunderstood techniques in applied AI. Here's why it works, where it helps, and where it's just theater.

Why "thinking out loud" helps a model

Recall that a model generates one token at a time, each conditioned on everything before it. When you force a straight-to-the-answer response, the model has to compute a complex, multi-step result in a single leap — and like a person blurting an answer without working it out, it makes careless mistakes.

Chain-of-thought (CoT) asks the model to generate the intermediate reasoning steps before the final answer. This isn't cosmetic. Each reasoning step the model writes becomes part of the context for the next step — so the model is literally building a scaffold it can lean on, breaking one hard leap into a series of easy ones. It gets to "show its work," and the work genuinely improves the destination.

There's a reason the newest "reasoning" models are, in essence, chain-of-thought industrialized: trained to think at length before answering, because more deliberate computation reliably produces better answers on hard problems.

Where it helps — and where it doesn't

CoT is not free, and it's not universal.

It helps on multi-step problems: math, logic, planning, debugging, anything where the answer depends on a chain of intermediate conclusions.
It barely helps on simple lookup or single-step tasks — "what's the capital of France" needs no scratchpad. Here CoT just burns tokens and latency for nothing.

The cost is real: reasoning steps are tokens, which means more money and more waiting. Use CoT where the problem is genuinely multi-step; skip it where the answer is direct. Matching technique to task is the whole game — the same judgment I apply across the systems I build.

From reasoning to acting

Chain-of-thought is also the seed of agentic behavior. Once a model can reason through steps, the natural next move is to let it act on those steps — use a tool, observe the result, reason again. "Think, then do, then think" is CoT with hands. The reasoning loop inside a single response becomes an action loop across many.

Using it well

Ask for steps explicitly on hard problems, and let the model finish reasoning before it commits to an answer.
Separate the reasoning from the final answer so you can extract the clean result — and inspect the reasoning when something looks off.
Don't force it everywhere. For simple tasks, it's overhead, not insight.

The lesson underneath all of it: an LLM reasons in the act of generating. Give it room to generate its way to an answer, and on hard problems it will reach a better one. More on how I use this at www.divyakush.com.