How Reasoning Models Work: From Chain of Thought to Test-Time Compute

Reasoning models write a long explanation before giving an answer, while an ordinary model answers straight away. It's tempting to call the first kind smarter because you can see more text. Sometimes the extra work helps; sometimes it just takes longer, or gives a mistake a convincing explanation. To understand how reasoning models work, we need to separate how they were trained, how much work they do per request, and whether the final answer passes a check we trust.

We'll cover those ideas, then run a small local experiment with Ollama's thinking controls. The test uses arithmetic because we can compute the expected answers ourselves instead of asking another model to judge. Four easy questions won't reproduce a research benchmark, but they turned up a surprise about what "thinking off" means. Download the code, or open reasoning.py, to follow along.

Start with chain of thought

Chain-of-thought prompting asks a model to produce intermediate reasoning rather than only a final response. That can help when several operations have to be connected, such as tracking quantities in a word problem. The intermediate text also becomes context that later tokens can use. It's still generated text, though: a sequence of plausible steps isn't a proof, and an early mistake can lead to a confidently explained wrong answer.

One way to use several reasoning attempts is to sample multiple solutions and compare their final answers. The self-consistency paper studies picking an answer from several sampled reasoning paths instead of trusting one. This spends more computation at inference time and helped under the paper's conditions. It doesn't make the attempts independent: several can share the same misreading of the question.

Training can reward answers that are checkable

A model can also be trained to do better on tasks with verifiable outcomes. In mathematics, a final number can be checked; in programming, a solution can be run against tests. The DeepSeek-R1 paper uses reinforcement learning for reasoning and reports behaviours such as revising an approach and checking a result. Those observations concern its own training setup. They don't mean every model with a thinking field shares that history.

Training changes learned parameters before you ask anything, which is the same idea behind fine-tuning with LoRA. Inference-time reasoning happens while answering one particular question. The two work together: training can make useful intermediate steps more likely, while your application decides how much computation to spend. So when an announcement promises "more reasoning", ask which one it means: a differently trained model, a longer response, several candidate answers, a verifier, or a mix. If the underlying transformer and attention mechanics are new to you, start there.

What test-time compute includes

Test-time compute is computation spent after training, while solving a particular task. Generating a longer attempt is one example; generating several candidates, running checks, revising a solution, or searching among partial solutions are others. The test-time scaling study by Snell and colleagues compares these strategies and shows that how compute is allocated matters. It isn't a rule that a bigger token budget improves every question.

Consider a hotel-policy question from a document the model has never seen. No amount of generated reasoning supplies the missing policy; retrieval or a trusted lookup does. A multi-step calculation with every quantity supplied is different, and might benefit from extra work plus an arithmetic check. The RAG tutorial covers fetching evidence, and the agent tutorial shows how to call an operation that returns a result.

A visible explanation isn't an internal audit

A model's written explanation can help you spot a bad assumption, but it isn't a complete account of what caused the answer. Turpin and colleagues show chain-of-thought explanations that rationalize answers without mentioning influences planted in biased prompts. Judge the final result on its own. For code, passing relevant tests is stronger evidence than a polished account of why it ought to work.

Likewise, an API field named thinking proves nothing about how a model was trained. It's part of the interface's output. Ollama documents thinking controls and response fields, and supported values vary by model: some take booleans, others named levels. Our experiment tests that interface with installed models; it can't inspect anything private about how they compute.

Design a test before changing the setting

We use four arithmetic questions with known integer answers. Each prompt asks for only the final integer, which gives us an explicit output contract to check instead of subjectively rating explanations. The code runs each question with thinking requested off and then on, and records elapsed time, generated tokens, the done_reason, and how many characters landed in each response field. A correct number surrounded by extra text fails this strict check on purpose.

CASES = [
    ("What is 17 times 23?", "391"),
    ("What is (18 + 7) times 4?", "100"),
    ("A box holds 24 pencils. Five full boxes lose 17 pencils in total. How many pencils remain?", "103"),
    ("What is 125 minus 3 times 17?", "74"),
]

Run the script from the starter folder after pulling the models; the Ollama setup guide covers installation. The first run uses the thinking model qwen3:4b. The second repeats the test with qwen3:4b-instruct, a model built to answer directly, as a baseline. Each run writes a JSON file you can inspect. Don't quietly extract whichever number matches the answer key from a long response, since that turns the checker into part of the solution.

ollama pull qwen3:4b
ollama pull qwen3:4b-instruct
unzip ai-blog-starters.zip
cd ai-blog-starters
python3 reasoning.py
BLOG_CHAT_MODEL=qwen3:4b-instruct python3 reasoning.py reasoning-results-instruct.json

Send the same question with two thinking modes

Within a run, the model, question, context size, temperature, and output limit stay fixed; only think changes. The 1,024-token generation limit keeps the experiment bounded. A reasoning attempt can use up that budget before writing an answer, so the code counts empty content as a failed match instead of assuming the request succeeded.

result = request_json("chat", {
    "model": "qwen3:4b",
    "stream": False,
    "think": think,
    "messages": [{
        "role": "user",
        "content": question + " Reply with only the final integer, no punctuation.",
    }],
    "options": {"temperature": 0, "num_ctx": 4096, "num_predict": 1024},
})
answer = result["message"]["content"].strip()
exact_match = answer == expected

Elapsed time includes more than reasoning: model loading and server queueing count too. Our script runs the modes in a fixed order with one repetition, so treat its timings as observations. A fair latency comparison warms up each configuration, alternates the order, repeats tasks, and reports variation.

What happened in our recorded run

We ran both commands on 2026-10-20 with Ollama 0.40.5 on a single NVIDIA GB10 GPU, using qwen3:4b (ID 359d7dd4bcda) and qwen3:4b-instruct (ID 0edcdef34593), at temperature 0, once per setting. The thinking-model results and instruct-model results are in the download.

Three things stand out. First, think: false didn't stop this model from reasoning. The reasoning simply arrived inside content (when it finished, it ended in a literal </think> tag before the number), and every question generated exactly the same number of tokens in both modes. The setting changed where the text went, not how much work was done.

Second, more reasoning has a failure mode of its own. On 17 times 23, both modes spent the whole 1,024-token budget without finishing. With thinking on, that meant an empty content field: no answer at all.

Third, the instruct model got all four right using 4 to 30 tokens, well under a second each once loaded. Two of its replies showed a line of working, so they fail the strict format check even though the arithmetic is correct. Explore every recorded response below.

None of this shows that reasoning models are useless. These are four easy questions, where long reasoning has little room to help; the research above reports gains on harder multi-step problems. What it does show is that "thinking off" depends on the model, that a token budget can end a reasoning attempt before it answers (an earlier recorded run of this script finished 17 times 23 in both modes, so results shift between runs), and that an exact-match score only means something when you can see what it counted.

When reasoning models are worth it

Test with tasks from the application you're actually building, and give each one an independent check. For coding, that means tests that exercise the required behaviour. For a document answer, it means finding the right evidence and checking the claim against it. For planning, it means checking dates, quantities, and resource limits. Extra reasoning earns its place only if it improves those outcomes enough to justify the latency and compute.

A practical application can spend different effort on different requests, but that routing needs evaluating too. An easy-looking question may hide an ambiguity, and a hard-looking one may just need a database lookup. Start by naming your failure types, then compare a few explicit strategies: a direct answer, a tool calculation, a bounded reasoning attempt. Keep the logs readable by a person, so you can tell when the extra work helped.