What Effort Level Should I Choose?
On the Economics of Reasoning Effort in LLM Agents

In economics, a credence good is an expertise service whose utility is difficult or impossible for the consumer to evaluate even after purchase—classic examples being automotive engine repairs, medical treatments, or complex legal advice. You pay an auto mechanic $1,200 for a replaced head gasket; the car drives afterwards, but did you truly need that specific repair, or would a $50 spark plug have solved the stall?

Reasoning effort dials (such as low, medium, high, xhigh in OpenAI's GPT-5.6 family or Anthropic's Claude Sonnet 5 adaptive thinking) are marketed as linear compute upgrades. However, users must choose this dial upfront, before the agent executes. If the task succeeds at high, the user has no way of knowing whether the run was successful because of the extra reasoning tokens or whether low would have succeeded for a fraction of the cost.

In multi-turn agentic workflows (e.g. SWE-Bench, Terminal-Bench), this upfront choice is amplified across dozens of sequential tool invocations. The fixed dial setting is multiplied on every step, repeatedly rereading an expanding context window and generating thousands of hidden tokens.

Standard benchmark reporting presents average accuracy scores, fostering the natural belief that solved sets are strictly nested:

In reality, this assumption fails routinely. When effort increases, models gain some previously unsolved instances, but simultaneously enter failure states on tasks that lower effort solved. The regression set $s_{e_1} \setminus s_{e_2}$ is non-empty for every single model evaluated on Terminal-Bench 2.1.

Does setting high effort allow the agent to quickly solve an easy task and terminate? No. On the intersection of tasks that are successfully solved across all effort levels ($s_{all}$), mean execution costs explode—up to 42× for Inkling, 4.54× for GPT-5.6-sol on SWE-Bench Pro, and 5.34× for GPT-5.6-luna.

Critically, this cost explosion is not caused by reasoning tokens alone. In agents, input-token charges can rise substantially as context grows (Figure 2; no universal percentage split is established here). When given high effort, models produce longer agentic trajectories with more tool invocations, each re-reading the entire accumulated conversation history.

Effort levels do not merely deliberate longer; they actively alter the problem-solving strategy. Using the Act·ONOMY agent behavior taxonomy across hundreds of annotated rollouts, the researchers discovered two major behavioral pathologies:

1. Git History Archaeology (Benchmark Leakage): On SWE-Bench Pro, repository history accidentally contained the gold reference fix commits. At low effort, GPT-5.6-sol examined git commits in only 15.9% of tasks. At xhigh, it aggressively ran git archaeology and read the reference-fix commit in 75.0% of tasks. When the authors created SWE-Bench Pro-Corrected (stripping git history and firewalling un-whitelisted internet access), maximum-effort solve rates fell across all 8 models (11.4–29.5 percentage points; statistically significant for 5 of 8) (e.g. GPT-5.6-sol fell from 77.3% to 59.1%, Claude Sonnet 5 from 79.5% to 50.0%).
2. Pathological Self-Verification: At higher effort, agents double and triple check their work (running test suites, diffs, linters). However, self-verification frequently checked conditions that diverged from external evaluators. In Terminal-Bench's dna-insert, the agent's high-effort run hallucinated that its output satisfied a constraint and halted, whereas its low-effort run completed cleanly. In sanitize-git-repo, GLM-5.2's max-effort run spent extra verification steps rewriting git history, inadvertently deleting a commit required by the external verifier!

Can models simply be prompted upfront to inspect the task prompt and declare: "This is simple, run at low" or "This is brutal, run at high"? The authors tested this Self-Routing policy across thousands of prompts. The outcome: self-routing yields no consistent advantage over naive random allocation. Upfront prompt inspection cannot forecast execution difficulty.

However, an astonishingly simple heuristic—Naïve Laddering—dramatically outperforms fixed settings:

Because laddering guarantees preservation of every task the low-effort model solves, its success set is the union $s_{\text{low}} \cup s_{\text{high}}$. As demonstrated in Figure 5 and Appendix H, naïve laddering matches or improves accuracy across all evaluated models and benchmarks while slashing median API bills by 25.3% on AIME, 41.5% on GPQA, and 15.5% on corrected SWE-Bench Pro!

Looking solely at provider API token bills fundamentally misranks model configurations because it ignores developer opportunity cost. The authors formulate the total economic cost $C_{\text{full}}$:

When factoring in human idle latency, waiting costs represent 58% to 98% of the true cost of running agentic workflows. Because naïve laddering solves the vast majority of tasks quickly at $e_{\min}$ without spawning massive reasoning chains, its full-cost advantage over fixed top effort is even greater (e.g. GPT-5.6-terra saves 19.5% in full cost vs only 4.8% in API charges alone).

• Single Agentic Rollout Constraint: Due to immense compute costs ($14,071.10 API spend in total), the primary Terminal-Bench and SWE-Bench Pro evaluations ran single rollouts per task-effort pair. While the authors verified that reversals persist under repeated 4-rollout executions on DeepSWE v1.1 (Appendix J), individual task regressions retain an element of stochastic sampling noise.
• Laddering Relies on an Oracle Verifier: Naïve laddering requires an external unit test or compiler feedback to decide whether to escalate. In open-ended customer support, literature synthesis, or exploratory coding where no test suite exists, laddering cannot easily trigger without autonomous self-evaluation (which currently suffers from severe hallucination).
• Proprietary Opacity: Frontier providers treat reasoning effort implementations as proprietary trade secrets. While open-weight models disclose continuous RL length-penalties or hard interruption thresholds, OpenAI and Anthropic do not disclose whether their settings alter temperature, system prompts, or internal test-time search policies.
• Multi-Tasking Opportunity Costs: The full-cost equation assumes users sit synchronously blocked during agent execution. In real workplaces, engineers switch contexts to other work (though research by Mark et al. and Leroy confirms substantial attention residue and cognitive restart penalties).