Stop Turning the Thinking Dial to High

When does a thinking product actually need to think?

When the cost of a wrong answer is high and the task needs real multi-step logic. That is the whole rule. Everything else is the expensive default most teams reach for without checking.

Reasoning models, the ones that spend extra inference compute on internal tokens before answering, are not a free upgrade. They are a trade. You spend latency and money to buy accuracy. That trade is excellent on a narrow band of hard, verifiable, high-stakes work, and it is a quiet loss on the majority of product interactions, which are lookups, classifications, and simple transforms. Building a good thinking product is mostly the discipline of sending thinking only where it pays.

How low can the bar for "worth it" sit? Lower than most people guess. An economic evaluation of LLMs on math problems found reasoning models offered a better accuracy-cost trade-off once the price of an error passed about $0.20 per query. Treat that as an order of magnitude from one benchmark, not a universal constant: the exact figure will shift with your task and your error costs. The useful takeaway is the shape of it. The break-even is cheap, which means the real question is rarely "is this task hard enough." It is "what does a wrong answer cost me here," and for a lot of product work, the honest answer is almost nothing. Autocomplete, tagging, routing, "summarize this in one line." On that work, reasoning is pure overhead.

What does the thinking dial actually cost?

More than the marketing pages admit. On simple queries, reasoning models generate seven to ten times as many tokens as non-reasoning models for the same accuracy, per Amazon's product team. The token bill is the obvious part. The latency is worse, because users feel every second.

The size of the trade is easy to see in a controlled case. Marco-o1, an early o1-style model, lifted accuracy on a math benchmark by 17 percent over its base model while increasing output tokens by 166 percent and inference time by 308 percent. The model is dated now, but the shape of the trade still holds: three times the wait for a real but modest accuracy gain. That is a fine deal on a hard problem and a terrible one on "what is 2 + 3," which some reasoning models answer only after generating hundreds of redundant tokens.

The mistake I see most often is treating reasoning as a global product setting. Teams flip the whole product to high reasoning out of caution, then pay several times over for workloads that never needed it, with a quality lift that does not show up in the numbers. The dial belongs on the request, not the product.

The overthinking trap: more thinking, worse answers

Here is the part that breaks the "thinking is always safer" intuition. Extra reasoning does not just waste tokens on easy tasks. It can lower the quality of the answer.

Researchers have documented an inverted-U relationship between chain-of-thought length and accuracy: accuracy climbs, peaks, then falls as reasoning runs long. Models exhibit what the literature calls overthinking, where they reach a correct answer and then talk themselves out of it over extended reasoning. A unified benchmark for over- and underthinking found cases where a model lands on the right answer early, then drifts to a wrong one because of conflicting self-generated information.

The practical lesson: on classification, filtering, and similarity tasks, long scratchpads make products less stable, not more. A team that switched a deduplication task from long reasoning back to short, constrained prompts saw accuracy recover once the model stopped inventing distinctions that were not there. If your task has a tight, well-defined output, give the model less room to think, not more.

The trust trap: visible reasoning is persuasion, not proof

The most popular pattern in thinking products is to stream the model's reasoning so users can watch it work. It feels transparent. The evidence says it leans closer to a persuasion device, and that should make you careful about how you ship it.

In a controlled hiring-decision study, showing the AI's reasoning raised user agreement with it from 88.7 to 90.7 percent. The shifts are small, a few points each. The striking part is the direction. When the model was explicitly missing a decisive piece of information the human held, agreement with it still rose, from 77.3 to 81.7 percent, after participants saw its reasoning. Revealing the thought process did not help people calibrate when to override the model. It nudged them to defer more, including when they should have trusted themselves. The extensiveness of the reasoning did not even matter; brief and long displays moved trust by the same amount.

There is a deeper reason to be cautious. A position paper argues that intermediate tokens are not guaranteed to carry the meaning users read into them, and that deliberately styling them to look like human deliberation engenders false confidence in answers users often cannot verify. Read that as a design warning. A streamed thought trace that looks thoughtful will make people believe a wrong answer. Showing reasoning is a feature you owe care to, not a trust meter to push to the top.

How to design the waiting, since you cannot remove it

If a task genuinely needs reasoning, the user will wait. The question is what the wait communicates.

Latency is more forgiving than dashboards suggest. In one study, user trust stayed high even at 20-second delays, and a moderate pause can raise the perceived usefulness of an answer because it reads as effort. But the same work warns that long delays inflate expectations: make someone wait, and the output had better justify it, or the gap between promise and result becomes the thing they remember. Do not buy perceived effort with a fake "thinking" spinner on a task where the extra time produced nothing.

A few patterns that hold up:

• Show real progress, not theater. A stepper that names actual stages (retrieving, drafting, checking) sets an honest mental model. A spinner pretending to deliberate over an instant lookup teaches users to distrust the next wait.
• Surface failure instead of silently retrying. "The lookup failed, retry or enter it manually" keeps the user in control.
• Replace history with a decision log: show why the product acted, not just that it did, so reasoning earns trust through accountability rather than spectacle.

A routing model you can ship

Match compute to the task. Amazon's team frames it as inflection points: requests that clearly need no extra thinking, requests that clearly do, and a grey zone in between. Build the router around those three.

Task	Example	Reasoning effort
Retrieval / lookup	account balance, capital of a country	None
Transform / classify	tagging, extraction, dedup	None to minimal
Grey zone	nuanced summary, ambiguous draft	Test both, route by eval
Multi-step, high cost-of-error	code fixes, reconciliation, planning	High

Set the thinking budget per workload using the controls the providers already expose: OpenAI's reasoning_effort parameter and Anthropic's configurable thinking budget do the same job through different dials. Then measure the only number that matters: cost per correct answer, not raw accuracy and not raw cost. An eval that mirrors production conditions will tell you, workload by workload, where the dial earns its rent and where it is burning budget for nothing.

The takeaway

A thinking product is not a product that thinks more. It is one that thinks exactly where thinking changes the outcome and stays fast and cheap everywhere else. The teams that win the next year will treat reasoning as a routed resource with its own telemetry, the way they already treat a database query or an API call, instead of a feature flag set to high and forgotten. Before you turn the dial up, make the model prove, on your own eval, that the wait buys you a better answer. Most of the time it will not.