Think Less, Locally: Local Preference Learning for Reasoning Compression

Modern reasoning models like DeepSeek-R1, OpenAI o1, and Kimi k1.5 have unlocked astonishing capabilities in competition math and code generation by generating thousands of hidden "Chain-of-Thought" (CoT) tokens. But this slow-deliberation superpower comes with a debilitating pathology: overthinking.

When asked a simple question like "What fraction of 1/2 is 1/7?" or "If July 3, 2030 is Wednesday, what day is July 14?", the paper shows examples in which a model has already derived and checked the correct answer, yet continues re-deriving it using modular arithmetic, calendar loops, or imaginary pizza slices. These examples illustrate redundant reasoning after the answer has been verified; they do not establish a typical token count or frequency.

LoCo executes three modular stages to extract self-supervised preference pairs directly from a model's own self-generated reasoning traces without requiring expensive manual annotations.

1. Step-Level Overthink Detection

Reasoning traces generated by DeepSeek-R1-Distill-Qwen-7B are parsed at paragraph boundaries into discrete steps $s_0, s_1, \dots, s_n$. The reasoning state after step $s_i$ is defined as the exact concatenation of the prompt and intermediate reasoning:

Candidate overthinking points are screened via two complementary intrinsic signals computed from the model's own next-token logprobs $p_\tau^{\max}$:

• Step Confidence $c_i$: The mean top-1 next-token probability over the span $T_i$ of step $s_i$: $$c_i = \frac{1}{|T_i|} \sum_{\tau \in T_i} p_\tau^{\max}$$
• Confidence Volatility $v_i$: A minimal 2-step rolling window measuring sudden fluctuations between adjacent steps: $$v_i = \frac{(c_i - c_{i-1})^2}{4}$$

A state is flagged if it exhibits simultaneously low confidence and high volatility: $$\mathcal{O} = \{i : c_i \le q_{25} \;\wedge\; v_i \ge q_{75}^v\}$$ This conjunction is critical: low confidence alone could indicate a hard problem, and volatility alone could mean a healthy transition. Together, they spotlight uncertain, wandering states.

Finally, a stronger LLM (Qwen3-30B) inspects $(q, P_i, s_{i+1})$ to filter out true derivation steps (NORMAL) or essential final answer restatements (FINAL_ANSWER_WRAPUP). Only confirmed OVERTHINK states survive.

2. Verified Continuation Generation & Local DPO

Once a state $i \in \mathcal{O}$ is confirmed, LoCo freezes prefix $P_i$. The model $M$ is then prompted to independently generate $k = 8$ continuations:

Any completion that fails to match the ground truth $z$ symbolically ($y \not\models z$) is discarded. From the pool of verified continuations $\mathcal{V}_i$, LoCo selects the shortest candidate:

If $|y_i^+| < |y_i^-|$ with a minimum token saving $\delta = 300$, a preference triple $(P_i, y_i^+, y_i^-)$ is formed and trained with Direct Preference Optimization (DPO):

Because both chosen and rejected branches share the exact same contextual prefix $P_i$, the gradient updates penalize the urge to deliberate further only when arriving at that state, avoiding across-the-board truncation that damages tough questions.

The authors trained DeepSeek-R1-Distill-Qwen-7B with LoRA on 14,027 math queries from math14k and evaluated on five math benchmarks (MATH-500, AMC 2023, Olympiad, AIME 2024, AIME 2025) and three coding benchmarks (MBPP, LiveCodeBench, CodeContests).

The results reveal two profound findings:

1. Harder problems benefit most: On the brutal AIME 2024 benchmark, LoCo cuts token count by 45.4% (from 13,493 to 7,364 tokens) while simultaneously boosting accuracy by +3.34 percentage points (50.83% → 54.17%). The comparison shows an accuracy–efficiency improvement; it does not by itself isolate a causal mechanism for individual errors.
2. Zero-shot cross-domain generalization: Despite training exclusively on mathematical traces, LoCo transferred to the three evaluated coding benchmarks, cutting 23.4% to 36.3% of tokens on Python tasks without losing accuracy, outperforming all compression baselines in Area Under the Curve (AUC).

Table 3 in the paper isolates every component in LoCo's architecture. What happens if you strip away the local state conditioning, change the generator model, or replace the confidence detector with random cutpoints?

While LoCo marks a major advancement in fine-grained reasoning compression, several open challenges remain:

• Judge Model Overhead: Candidate overthinking steps must be verified by a larger model (Qwen3-30B) to distinguish between harmless repetition, productive intermediate exploration, and final formatting. Creating an entirely judge-free, self-contained detector remains an open question.
• Causal Truncation vs Intermediate Editing: LoCo freezes prefix $P_i$ and discards the entirety of the subsequent original trace $y_i^-$. If a model suffers from local overthinking in step 4 but needs steps 6–10, LoCo relies on its regeneration step to re-derive the remaining steps succinctly rather than performing surgical in-place edits.
• Scalability to Frontier Models: Experiments were demonstrated on DeepSeek-R1-Distill-Qwen-7B. It remains to be seen whether 70B+ scale frontier models exhibit identical confidence volatility profiles before overthinking loops begin.