The Limits of a Single Prompt: Why Reasoning Doesn't Guarantee Precision

The more a model "thinks" before answering, the higher the chance the answer will be correct—an assumption that is easy to accept uncritically. We tested it empirically on three tasks requiring mechanical, verifiable precision, and the result is more complex than it might seem.

What is a Single Prompt?

A single prompt is the simplest possible way to use a language model, exactly the one most of us know from everyday conversations with ChatGPT: we write a question or command, the model generates one answer, and that's it. The model has no way to check if what it just wrote is correct, it doesn't use a calculator, it doesn't run any code, or consult with itself. It just predicts the next word, then the next, until it finishes the sentence, and whatever comes out goes to us as a ready answer.

This approach works great in conversations, writing texts, or summaries where there is no single "hard" correct answer. The question we asked ourselves was: what happens when a task has exactly one correct answer and requires mechanical precision, like the result of multiplication or the exact number of characters in a text? Does the mere fact that we tell the model to "think longer" (increase the reasoning_effort) guarantee such precision?

The tests were run via Azure OpenAI, on a GPT-5.6 family model. The reasoning_effort parameter controls how many "thinking" tokens the model consumes before generating an answer, from none to xhigh. To distinguish systemic problems from sampling noise, we repeated each task × effort combination four times.

Three Tasks, One Common Feature

We chose three tasks with results that are either strictly correct or incorrect:

• Multiplication — product of two 4-digit numbers, exact result.
• Characters — a text with exactly a specified number of characters (not words).
• Letters — counting the occurrences of a specific letter in a random, ~350-character text.

Results: Success Rates Grow with Effort, But Not Linearly

At effort=none, all three tasks fail in 100% of the attempts. What's surprising is what happens next. For the Letters task, high effort (75%) performs worse than low and medium (100%), so more reasoning doesn't mean a better result. Higher effort increases the probability of hitting the mark, but doesn't guarantee it.

Why Does This Happen?

At effort=none, the model was supposed to calculate 3471 × 6305 = 21,884,655. It returned: "218637?", a number that was not only wrong but structurally incomplete. This illustrates the mechanics: the model generates an answer token by token, one way, without verifying its own output. There is no "check if this adds up" step. Reasoning effort gives the model more "space" to break down the problem before answering, but it doesn't change the fundamental architecture: it's still one-way generation without a feedback loop.

Cost Grows Faster Than Certainty

Higher reasoning effort means higher costs—we pay for the "thinking" tokens, not just the answer.

For the Characters task, the cost grows from $0.0003 (none) to $0.0142 (xhigh) — roughly 40 times more. The success rate at xhigh actually reaches 100%, but at high it's only 50%. We pay more, not always buying more certainty.

Letters: average single-prompt cost

Reasoning effort	Average cost (USD)	Success rate
none	$0.000249	0%
low	$0.003133	100%
medium	$0.004803	100%
high	$0.005837	75%
xhigh	$0.006423	100%

Characters: average single-prompt cost

Reasoning effort	Average cost (USD)	Success rate
none	$0.000342	0%
low	$0.010084	0%
medium	$0.010458	50%
high	$0.014829	50%
xhigh	$0.014225	100%

What's Next

Since the problem lies in the lack of verification, the natural step is to give the model a tool that verifies for it. In the next post, we check what code execution gives and where it still falls short.