Test-Time Compute: The Model Decides Your Bill

Key takeaway: Every serving optimization in this series makes a known number of output tokens cheaper. Reasoning models break that: they spend output tokens thinking, those tokens bill at the output rate (roughly 5x input), and the model picks how many. The decision this post helps you make: cap and price the length the model chooses, not the one you assumed.

Quantization, continuous batching, speculative decoding, paged KV: all of them optimize a known output-token count. A reasoning model makes that count a variable it sets per request.

DeepSeek published the shape of it. Across a set of 2024 competition math problems, R1 averaged 8,793 thinking tokens per problem, spending fewer than 7,000 on the simple ones and more than 18,000 on the hardest (arXiv:2501.12948, Supplementary Information, Figure 18). Nobody set those lengths. The model allocated them per question.

That makes output length a long-tailed distribution, and both your bill and your concurrency ceiling follow from it.

Previous post: The KV Cache Is Your Concurrency Ceiling.

What this post covers

• What test-time compute is, and how it left your control
• The scaling reframe: buying accuracy with inference compute instead of parameters
• How reasoning tokens bill, and why that is the expensive side
• What variable output does to memory and concurrency
• The controls that ship today, and what to log
• Why this one cuts against the usual serving-versus-context story

Test-time compute is accuracy bought at inference, not at training

Test-time compute is computation spent at request time to raise quality: tokens generated before the answer, where the model works the problem out. Accuracy scales with inference tokens, not just trained parameters.

Three steps loosened your grip:

1. Chain-of-thought prompting. A 540B model with eight worked exemplars reached state of the art on GSM8K, beating a finetuned GPT-3 with a verifier (Wei et al.). You wrote the reasoning.
2. Self-consistency. Sampling many reasoning paths and keeping the most consistent gained GSM8K +17.9%, SVAMP +11.0%, AQuA +12.2% (Wang et al.). The sample count was yours.
3. Trained-in reasoning. DeepSeek-R1-Zero's AIME 2024 pass@1 rose from 15.6% to 77.9% through pure reinforcement learning, and to 86.7% with self-consistency decoding, the same trick from step 2. What you used to configure is now a model behavior.

Inference compute substitutes for parameters, conditionally

Snell et al. titled it "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters" (arXiv:2408.03314). Allocating that compute well improves "the efficiency of test-time compute scaling by more than 4x compared to a best-of-N baseline." FLOPs-matched, "on problems where a smaller base model attains somewhat non-trivial success rates, test-time compute can be used to outperform a 14x larger model."

Keep the conditional clause; the 14x is not universal. The point is the substitution: fixed training cost becomes variable per-request cost.

Reasoning tokens bill as output, and output is the expensive side

OpenAI is explicit: reasoning tokens are invisible in the response but "still occupy space in the model's context window and are billed as output tokens." Anthropic bills thinking tokens as output too: you pay "for the full thinking process, not the thinking content visible in the response."

Across the list prices below, output runs four to six times input, and reasoning tokens land entirely on that side.

There is a worse case than an oversized invoice. Hitting max_output_tokens returns status: incomplete, and "you could incur costs for input and reasoning tokens without receiving a visible response." You can pay in full and get nothing back.

The published counts are wide enough to plan around, but only just. OpenAI says its models "may generate anywhere from a few hundred to tens of thousands of reasoning tokens." DeepSeek's own comparison is sharper: on the same math problems where R1 spent an average of 8,793 thinking tokens to reach a 61.8% solve rate, GPT-4o 0513 generated 711 output tokens on average for a 24.7% solve rate. That is at least a twelvefold difference in billed output for one swap of model class, since R1's 8,793 counts only its thinking and the answer bills on top. The accuracy you buy with it is real. Both numbers are averages over a benchmark, not a forecast for your traffic.

Variable output lowers your concurrency ceiling twice

From the previous post's KV formula (arithmetic, not measurement):

bytes = 2 × layers × kv_heads × head_dim × seq_len × bytes_per_element

seq_len counts generated tokens, so a request that thinks ten times longer holds roughly ten times the cache for ten times as long: the ceiling falls on both axes at once. Decode is memory-bandwidth-bound, so those tokens bill at the expensive per-step rate. Prefill is unchanged, so TTFT barely moves; the unbounded part is all in decode.

An empirical study of reasoning-model serving reports four behaviors (Li et al.): memory fluctuation, straggler requests, adaptive running time, and domain preference. Read that last one as cost varying by topic and not only by user, which is our reading of the term rather than the authors' definition. It also finds prefix caching and KV-cache quantization can degrade accuracy or serving performance for small reasoning models, while quantization and speculative decoding survive the transition cheaply. We have recommended that first pair in earlier posts, so revalidate both against reasoning traffic before trusting them there.

The controls that ship today

You cannot set the reasoning length, only bound and observe it.

• Effort levels. OpenAI's reasoning.effort runs none to max; Anthropic's output_config.effort runs max to low, where Claude "skips thinking for simple tasks." Routing now lives inside the model, changing the build-versus-configure call.
• Budgets are targets, not caps. Anthropic's budget_tokens starts at 1,024, sits below max_tokens, and the docs call it "a target rather than a strict cap." Deprecated on Claude 4.6 and rejected on 4.7, it leaves max_tokens as the only hard cap.
• Top-level effort changes invalidate the prompt cache. The resolved effort value is rendered into the prompt, so varying it between requests costs you your prompt caching savings. There is a carve-out worth knowing: on Claude Opus 5, Fable 5.1, and Mythos 5.1, a per-message effort change preserves the cache, currently behind a beta header. Everywhere else, pick a level per conversation and hold it.
• Leave room, or pay for nothing. Thinking counts against max_tokens, so a cap sized for a no-thinking answer truncates once the model starts reasoning. OpenAI suggests reserving at least 25,000 tokens for reasoning plus output when you begin.
• Log the real count. Alert on the p99 of reasoning_tokens and thinking_tokens in usage.output_tokens_details, not the mean. Once output length varies, cost per solved task is the only honest comparison.

The bottom line, and where to start

Two axes: the serving layer buys throughput and cost, the context layer buys correctness and freshness. Test-time compute is the first to cross that line the wrong way. Quantization makes you wrong faster and cheaper; continuous batching serves the stale answer to more people for less; a reasoning model makes a wrong answer more expensive than a right one. Stale retrieval is not caught by longer thinking; it is elaborated at the output rate.

Start here: log reasoning_tokens on your busiest endpoint for a day, then set max_tokens from the p99. If that tail is spent reconciling contradictory chunks, the fix is upstream: get started in the docs. Measured a surprising tail? DM Flow on X with the ratio.

Next read: Continuous Batching Is How One GPU Serves a Crowd, the throughput lever that assumes a fixed output length.