Reasoning Models Explained: When Chain-of-Thought Pays Off

A reasoning model is an LLM post-trained to spend extra compute generating an internal chain of thought before it commits to an answer. You are billed for those thinking tokens at output-token rates, so the real question is never “is it smarter?” — it is whether the accuracy gain on this task justifies the added cost and latency.

This guide covers the mechanics, the API-level differences that bite in production, a routing table, and a measurement method you can run on your own traffic in an afternoon.

What a reasoning model actually is

A standard instruct model is trained to map a prompt to an answer directly. Supervised fine-tuning and preference tuning teach it to follow instructions and emit the response immediately, one token at a time. The tokens you see are the computation.

A reasoning model shares the same transformer backbone but is post-trained differently: reinforcement learning against a verifiable reward — a math answer that checks out, a unit test that passes, a constraint that is satisfied. That training pressure teaches the policy to generate a long internal trace before producing the final answer, because longer deliberation measurably raises the reward on hard problems. This is test-time compute: accuracy scales with thinking length, not with parameter count.

Four consequences follow, and each one changes how you should call the API:

• Thinking is a budget, not a switch. Most providers expose an effort level or an explicit thinking-token budget. Low effort is a different product from high effort: same model, very different cost and latency.
• Chain-of-thought prompting is not the same thing. Asking any model to “show your steps” produces a trace. A reasoning model was trained to produce one, and the raw trace is frequently withheld or replaced with a summary.
• The trace is not an audit log. Visible reasoning can be unfaithful to the computation that actually produced the answer. Do not build compliance or debugging workflows on the assumption that the shown steps are the real ones — verify the answer instead.
• Scaling is not monotonic forever. Accuracy rises with thinking length up to a plateau, and on some tasks overthinking degrades it. “Max effort everywhere” is not a strategy.

How they differ at the API level

This is where most teams get surprised, because the differences are not just quality. They are billing, latency shape, context accounting, and streaming behaviour.

Thinking tokens are output tokens. In OpenAI-shaped responses, usage.completion_tokens includes them, and usage.completion_tokens_details.reasoning_tokens breaks them out. In Anthropic-shaped responses they arrive as separate thinking blocks. Either way: a 300-word answer can bill several thousand output tokens, and the ratio varies per request because the model decides how long to think. Your p95 cost can be several times your p50.

Reasoning may or may not be returned. Three patterns exist in the wild: full raw trace, a model-written summary, or nothing at all. Write your parsing code to tolerate all three rather than assuming a reasoning_content field exists. If the trace is absent, you cannot log it, so do not build an eval that depends on it.

Latency shape changes, not just latency. Time-to-first-token is often fine, but time-to-answer can be 3–20× longer, because nothing user-visible is emitted while the model thinks. In a streaming UI that is dead air: emit a heartbeat or an explicit “reasoning” state, or users will assume the request hung.

Context accounting is easy to get wrong. Thinking tokens occupy the context window for that turn. In multi-turn agent loops, some APIs require you to pass reasoning items back verbatim so the model can continue coherently; if you strip them to save tokens, quality can silently drop with no error. Check your provider’s contract before you optimise it away.

Retries are expensive. A timeout that triggers a retry re-bills the entire thinking phase. Set client timeouts above your observed p99, and never set a timeout shorter than the thinking budget you asked for.

Here is the smallest correct call pattern for both modes, with the billing arithmetic made explicit:

IN_RATE, OUT_RATE = 1.0, 1.0   # use your provider's relative rates

def call(prompt, mode):
    """mode='fast' -> plain instruct model; mode='reason' -> reasoning model."""
    if mode == "fast":
        r = client.chat.completions.create(
            model="fast-instruct-x",
            messages=[{"role": "user", "content": prompt}],
            temperature=0)
    else:
        r = client.chat.completions.create(
            model="reasoning-model-x",
            messages=[{"role": "user", "content": prompt}],
            reasoning={"effort": "medium"},        # shape varies by provider
            max_completion_tokens=8192)            # cap = cost + latency guard

    u = r.usage
    hidden = getattr(u.completion_tokens_details, "reasoning_tokens", 0) or 0
    cost = u.prompt_tokens * IN_RATE + u.completion_tokens * OUT_RATE

    # Log this per request: it is the only way to see the p95 blowup.
    log(mode=mode, visible=u.completion_tokens - hidden,
        thinking=hidden, cost=cost)
    return r.choices[0].message.content

The thinking field is the number to watch. It is also the input to every routing decision below — you cannot optimise what you do not measure. If you want the wider cost toolkit around caching, batching, and token budgeting, our reduce AI API costs guide covers it.

When reasoning pays off — and when it is pure waste

The useful filter is not “hard vs easy.” It is two questions: is the answer verifiable, and do errors compound?

Task type	Recommended mode	Why
Math / arithmetic with a checkable answer	Reasoning, medium–high effort	Correctness is verifiable and errors are costly downstream
Multi-file debugging, root-cause analysis	Reasoning, high effort	Several constraints must be held simultaneously
Algorithm design, competitive programming	Reasoning, high effort	Unit tests give you a free, objective reward signal
Multi-step planning, long-horizon agent loops	Reasoning, medium effort	A wrong step early poisons every later step
Constraint satisfaction (scheduling, config, allocation)	Reasoning, medium–high effort	Combinatorial search, not recall
Analysis where the answer must be computed from a table	Reasoning, medium effort	Thinking longer genuinely helps compute
Schema-constrained extraction from a clean document	Fast instruct	Pattern matching; ambiguity is in the schema, not the reasoning
Classification: intent, sentiment, spam, moderation	Fast instruct	Short, high-volume, low per-item cost — latency dominates
Summarisation, rewriting, translation	Fast instruct	Style task with no verifiable ground truth
RAG answer with one clearly relevant passage	Fast instruct	The answer is already in the context
Chit-chat, FAQ with a known answer	Fast instruct	Latency is the product
Creative ideation, divergent copy variants	Fast instruct	Deliberation converges on the safe answer and kills diversity

Two non-obvious traps hide in that table.

Reasoning cannot fix a knowledge gap. If the model does not know the fact, thinking longer will not retrieve it — it will produce a more elaborate wrong answer. When accuracy stalls on a fact-heavy task, add retrieval (see embeddings and RAG) instead of raising the effort level. Escalating effort is the reflex; it is usually the wrong lever.

Cheap-to-detect errors should never be paid for up front. If a wrong answer fails a unit test, breaks a JSON schema, or misses a numeric tolerance, run the fast model first and escalate only on failure. Verification is free; escalation is rare. That single pattern captures most of the reasoning model’s accuracy at a fraction of its cost, and it is the backbone of the hybrid router further down.

Cost and latency: measure the delta, not the absolute

Model prices move constantly, so reason in ratios. The per-request cost is:

cost = prompt_tokens * input_rate
     + (visible_output_tokens + reasoning_tokens) * output_rate

Three stable observations about that formula:

• Effective cost per request on a reasoning model is typically 3–15× a fast instruct model. The multiplier is driven mostly by thinking tokens, not by the base per-token rate.
• The variance matters more than the mean. Because thinking length is decided per request, p95 request cost can be several times p50. Budget on p95, not on average.
• Latency penalty lands on the tail. A reasoning model may match the fast model on time-to-first-token while being several times slower to a finished answer.

The metric that decides the trade-off is cost per correctly completed task (CPCT): total billed cost across all attempts — including retries and escalations — divided by the number of correct answers. Per-token price is a distraction. A cheaper model that is right 70% of the time and pushes 30% of requests into human review is usually the more expensive system.

Guardrails worth setting on day one: a per-call-site thinking budget, a max_completion_tokens cap that includes reasoning, a per-user daily spend ceiling, and an alert when p95 request cost moves more than 2× week over week. That last alert catches prompt rot before it reaches your invoice.

Prompting reasoning models correctly

Prompts tuned for instruct models actively hurt reasoning models. The model already has a trained deliberation policy; your job is to give it a clean problem, not a procedure.

• Give a spec, not a recipe. State the goal, the constraints, the inputs, and the exact output contract. A hand-written step list constrains the search space the model was trained to explore.
• Do not add “think step by step.” It is redundant at best. At worst it pushes the model toward a shallower, more formulaic trace than its trained policy would have produced — and it burns prompt tokens on every call.
• Do not paste few-shot CoT exemplars. Hand-written reasoning examples teach a worse trace. If you need examples, provide input→output pairs only and let the model derive its own path.
• Remove conflicting instructions. “Answer in one word” plus a hard math problem, or “be concise” plus a multi-constraint plan, forces the model to trade off two goals you did not intend to make mutually exclusive. The visible symptom is erratic thinking length.
• Set effort per call site, not globally. Extraction-shaped calls get low effort; verification-shaped calls get high. A single global default means you overpay on the easy traffic and underperform on the hard traffic.
• Define the stopping condition for agentic loops. Reasoning models will happily keep planning. A hard step cap and an explicit “stop when X is true” prevents a runaway bill.

The difference is stark in practice:

# Weak: procedure + conflict + no output contract.
bad = """You are an expert. Think step by step and reason carefully.
Be concise. Respond in exactly one word if possible.
Here are examples of how to reason: 1) First I ... 2) Then I ...
Question: which deployment window satisfies all constraints?"""

# Strong: goal + constraints + output contract. Nothing else.
good = """Pick the deployment window that satisfies every constraint below.
If no window satisfies all of them, return the single best-effort window
and list the constraints it violates.

Constraints:
- region: eu-west-1 only
- freeze: no deploys 2026-12-20..2027-01-03
- minimum 2 on-call engineers present
- DB migration must run at least 4h before the app deploy

Return JSON: {"window_start": ISO8601, "window_end": ISO8601,
"violations": [string]}"""

Note what the strong prompt does not do: it never mentions reasoning. The output contract is explicit, the constraints are enumerated so the model can check them one by one, and there is exactly one goal. That structure is what makes the thinking productive.

Hybrid routing: reasoning for the hard 15%

You do not choose between a reasoning model and a fast model. You route between them, and you make escalation conditional on a cheap deterministic check. The best router is your own call site: your application already knows whether it is doing an extraction or a plan, so tag the task explicitly instead of asking a classifier to infer it. For the wider framework, see our guide to model routing.

import json

FAST, REASONING = "fast-instruct-x", "reasoning-model-x"

def validate(text):
    """Free, deterministic check. Replace with tests / schema / tolerance."""
    try:
        data = json.loads(text)
    except json.JSONDecodeError:
        return False
    return {"window_start", "window_end", "violations"} <= set(data)

def answer(prompt, hard=False):
    # Explicit segment tag beats an inferred one.
    if hard:
        return _reason(prompt, effort="high"), "reasoning"

    # Cascade: cheapest model first, escalate only on verification failure.
    r = client.chat.completions.create(
        model=FAST, temperature=0,
        messages=[{"role": "user", "content": prompt}])
    text = r.choices[0].message.content
    if validate(text):
        return text, "fast"

    return _reason(prompt, effort="high"), "reasoning-escalated"

def _reason(prompt, effort):
    r = client.chat.completions.create(
        model=REASONING,
        reasoning={"effort": effort},
        max_completion_tokens=8192,
        messages=[{"role": "user", "content": prompt}])
    return r.choices[0].message.content

Two operational details make or break this pattern. First, log the route and the thinking-token count on every request — escalation rate is your leading indicator, and a rising rate usually means your prompt drifted rather than that your traffic got harder. Second, if the escalation rate exceeds roughly 20%, fix the prompt or tighten the validator before adding budget; a validator that rejects good output turns your cheap path into a dead weight you pay for twice.

How to evaluate the trade-off on your own workload

Public benchmarks answer the wrong question. Run a paired comparison on your own traffic:

• Build a labelled set from real traffic — 50–200 items, stratified by difficulty, each with a ground truth or a deterministic checker. Benchmarks measure general capability; this measures your task.
• Hold everything constant except the model. Same prompt, same temperature (0 for reproducibility), same item order. Any other change invalidates the comparison.
• Grade deterministically where you can. Exact match, numeric tolerance, unit tests, schema plus field-level assertions. Use an LLM judge only for open-ended output, and hand-check ~20 items to estimate the judge's own error rate.
• Report a 4-tuple, not a single number: accuracy, cost per correctly completed task, p95 latency, and escalation rate.
• Decide per segment. The aggregate hides the win. If reasoning only helps on the hardest 15% of traffic, routing by segment gets you nearly all the accuracy for a small slice of the spend.
• Re-run on every prompt change, model version bump, or traffic-mix shift. Keep the eval set in version control next to the prompt.

Reading the results is mechanical once you have the 4-tuple. For the broader benchmarking methodology, see evaluating AI models.

Eval result	What it means	Decision
Accuracy +under 2 pts, CPCT 5× or more	The task is knowledge-bound, not reasoning-bound	Keep the fast model and add retrieval
Accuracy +under 2 pts across every segment	Deliberation adds nothing here	Route the whole task to the fast model
Accuracy +10 pts overall, +25 pts on the hard segment	Gains are concentrated, not uniform	Segment-route: reasoning only for hard items
Accuracy +8 pts but p95 latency 4×	Correct, but unshippable on an interactive path	Move to async or batch, or drop to low effort
Accuracy flat and CPCT lower	Escalation is firing far too often	Fix the validator or tighten the prompt first
Accuracy +6 pts, CPCT only 1.4×	Reasoning tokens are short and the task is genuinely hard	Ship it as the default for that task

Frequently asked questions

Do reasoning models always give better answers?

No. They win where correctness is verifiable and errors compound — math, debugging, planning, constraint solving. On classification, extraction, summarisation, and conversational replies they are usually a more expensive way to get the same answer, and on creative tasks their tendency to converge on the safe answer can make output worse.

Are thinking tokens billed if I never see them?

Yes. Reasoning tokens are output tokens and are billed at the output rate whether the provider returns the trace, returns a summary, or returns nothing. They also count toward completion_tokens, which is why a short visible answer can produce a large bill. Always read the reasoning-token count out of the usage object and log it.

Should I add "think step by step" to a reasoning model?

No. The model was trained to deliberate, so the instruction is redundant and can push it toward a more formulaic trace than its policy would produce. Spend those prompt tokens on a precise output contract and an enumerated constraint list instead — that is what actually raises accuracy.

Can I use a reasoning model for a streaming chat UI?

Only with care. If the provider does not stream thinking, users see dead air for the whole deliberation phase even though time-to-first-token looks healthy. Either emit a heartbeat or an explicit "thinking" state, use a low effort budget on the interactive path, or route interactive turns to a fast model and reserve reasoning for an async job. Our streaming and SSE guide covers the wire format and the proxy-buffering trap.

Conclusion

Reasoning models are a compute-for-accuracy trade, not an upgrade. They pay off when the answer is verifiable and a wrong answer is expensive to detect; they are waste when the task is pattern matching, style, or conversation. The engineering work is therefore not "pick the best model" but three habits: log thinking tokens per request, run a paired eval to get cost per correctly completed task, and route by segment with a free deterministic check before escalating.

Put those habits behind a single OpenAI-compatible endpoint and the whole thing becomes a configuration change rather than a refactor — one base URL, one key, and a different model string per route. qoraapi.com exposes reasoning and fast instruct models through one endpoint, which is what makes the hybrid router above practical to ship and cheap to re-tune.

Related reading

• How to Choose the Right AI Model: A Practical Model-Routing Guide
• Evaluating and Benchmarking AI Models Before You Ship
• How to Reduce AI API Costs: A Practical Guide for Developers
• Local LLMs vs API: A Real Cost and Latency Comparison for 2026
• AI Gateway vs API Gateway: Key Differences and When to Use Each
• Top 10 Real-World Use Cases for an AI API in 2026