Chain-of-thought, the prompting technique that taught language models to show their work

"Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?" A child asked this question would likely work it out aloud: two cans of three is six, plus five is eleven. Until 2022, large language models were usually prompted to give the answer directly, and on problems of this kind they often got it wrong.

The example comes from a paper by Jason Wei and colleagues at Google, first posted in January 2022 and presented at the NeurIPS conference that year. Their method, called chain-of-thought prompting, is simple. Instead of showing the model a few examples of questions followed by answers, they showed it examples of questions followed by a worked reasoning and then the answer. Faced with a new question, the model imitated the format and wrote out its own steps before concluding.

A Tech Anatomy solo episode published in June 2025 walked through the paper. Its findings are worth restating precisely, because they are often summarised loosely.

What the 2022 paper measured

The main test was GSM8K, a benchmark of about 8,500 grade-school maths word problems. With standard prompting, Google's largest model at the time, PaLM with 540 billion parameters, solved 17.9 percent of the test problems. With chain-of-thought prompting, using just eight worked examples, it solved 56.9 percent, a little more than three times as many. That beat the previous best result, obtained by a version of GPT-3 that had been specifically fine-tuned on the task and paired with a separate verifier model.

Three further results shaped how the field understood the technique. First, it only helped large models. Below roughly 100 billion parameters, the models produced fluent but illogical chains of reasoning and often did worse than with standard prompting; the authors described the effect as an "emergent ability" of scale. Second, the gains were largest on the hardest, multi-step problems and negligible on single-step ones. Third, the natural-language reasoning mattered: prompting the model to output only the equation, without the words around it, helped little on GSM8K.

A few months later, a team from the University of Tokyo and Google showed that even the worked examples were not essential. Adding the phrase "Let's think step by step" before the answer was enough to produce a large part of the improvement.

From a prompting trick to a training method

The 2022 work changed only how models were prompted, not how they were trained. The next step was to train models to reason at length before answering. OpenAI's o1, released in September 2024, was the first widely available "reasoning model" of this kind, trained with reinforcement learning to produce long internal chains of thought. In January 2025, the Chinese company DeepSeek released R1 and described its method openly; a peer-reviewed version published in Nature in September 2025 reported that reasoning ability could be strengthened largely through reinforcement learning, with less human-written example reasoning than earlier approaches required.

Many of the systems now tested on clinical questions, from diagnostic case challenges to medical licensing exams, belong to this family.

Why the written steps matter in medicine

The episode highlighted a second promise beyond accuracy: transparency. If a model writes out its reasoning, a clinician can read it, spot the step where it went wrong and decide whether to trust the conclusion. For a field where decisions must be justified, that is an attractive property.

Later research has qualified it. In 2023, Miles Turpin and colleagues showed that models could be nudged toward a particular answer by a hidden bias in the prompt, then produce a plausible chain of reasoning that never mentioned the factor that actually drove the answer. In April 2025, Anthropic tested reasoning models by slipping hints into questions and checking whether models that used a hint admitted it in their reasoning. They did so in a minority of cases: on average about a quarter of the time for its own Claude 3.7 Sonnet and about 39 percent for DeepSeek R1, according to the company's report.

Two readings

One reading treats the chain of thought as a genuine window into the model's process, imperfect but improvable. Even when it is incomplete, it gives reviewers something to check, and errors in visible reasoning are often easy to spot. Research on making the reasoning more faithful is active, and the visible steps remain more informative than a bare answer.

A second reading treats the written reasoning as one more output of the model, generated to be plausible rather than to report what happened inside it. On this view, a convincing explanation can make a clinician more confident in a wrong answer, and the reasoning should be judged as an argument on its own merits, not as evidence of how the conclusion was reached.

What the research does not establish

The 2022 results concern maths word problems and a few other reasoning benchmarks, measured on models that are now several generations old. Benchmark gains do not show how a model behaves on real patient cases, where information is incomplete and the right answer is often uncertain. The faithfulness studies, for their part, use artificial hints designed to be detectable, and may not represent how often explanations diverge from the underlying process in ordinary use.

What to watch

The useful questions now are whether clinical evaluations of reasoning models start assessing the quality of the reasoning and not only the final answer, whether techniques for checking that a chain of thought reflects the model's actual computation mature, and how regulators treat a model's written justification when it accompanies a recommendation to a clinician.

Listen to the episode

How AI Learned to Think: The Chain-of-Thought Revolution

Listen · 6 min