Kimi k1.5: Scaling Reinforcement Learning with LLMs

Why This Matters

Pretraining LLMs via next-token prediction is hitting a data wall. High-quality human text is finite, and continuing to scale models simply by feeding them more static data is yielding diminishing returns. Scaling Reinforcement Learning (RL) offers a new axis of improvement: instead of memorizing static facts, models learn to explore, self-correct, and reason through trial and error.

However, state-of-the-art reasoning models like OpenAI's o1 are incredibly complex to train. Many researchers believe that achieving competitive performance requires heavy machinery: Monte Carlo Tree Search (MCTS), explicit value functions, and dense Process Reward Models (PRMs). Kimi k1.5 challenges this consensus by showing that a surprisingly simplistic RL framework, when paired with massive context scaling (up to 128k) and robust policy optimization, can match or exceed world-class performance.

The Big Idea

The core thesis: Rather than explicitly building search trees at inference time, we can train the model to perform an implicit search directly within its auto-regressive text output (its Chain of Thought). By scaling the context window to 128k tokens during RL, the model naturally develops planning, reflection, and error-correction capabilities.

"An increased context length has an effect of increasing the number of search steps. As a result, we show that strong performance can be achieved without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models."

How It Works

1. Policy Optimization via Online Mirror Descent

Kimi k1.5 uses a variant of Online Mirror Descent (OMD) for robust policy optimization. Instead of standard PPO, which requires a separate, fragile value network for credit assignment, k1.5 optimizes a relative entropy regularized objective:

$$\max_{\theta} \mathbb{E}_{(x,y^*) \sim \mathcal{D}} \left[ \mathbb{E}_{(y,z) \sim \pi_{\theta}} [r(x, y, y^*)] - \tau \text{KL}(\pi_{\theta}(x) || \pi_{\theta_i}(x)) \right]$$

Where $\tau$ controls regularization. This objective has a closed-form solution, which allows the team to derive a surrogate loss that can leverage off-policy data during optimization. Crucially, they skip the value network entirely, allowing the model to learn trial-and-error patterns directly from final outcome rewards rather than penalizing intermediate exploratory mistakes.

1

Interactive Loss & Gradient Explorer

Adjust variables to see how the Relative Entropy Regularized gradient update scales. Watch how the KL penalty prevents the policy from drifting too far from the reference model $\pi_{\theta_i}$.

Reward $r(x, y, y^*)$ 1.0

1.0 = Correct answer, 0.0 = Incorrect

KL Regularization $\tau$ 0.5

Higher values penalize deviation from reference policy

Log Policy Ratio $\log \frac{\pi_\theta}{\pi_{\theta_i}}$ 0.2

Positive means current policy is more confident than reference

Calculated Gradient Step

0.00 arbitrary units

The policy receives a positive update to reinforce this trajectory.

Penalize / Suppress No Change Reinforce / Boost

Loss Component: 0.00

2. Scaling Context to 128k via Partial Rollouts

Generating long thinking trajectories (long-CoT) is incredibly expensive. If a model generates up to 128,000 tokens of reasoning, standard RL rollouts require generating the entire sequence from scratch at every iteration. This creates massive GPU idling and bottlenecks.

To solve this, Kimi k1.5 introduces Partial Rollouts. Instead of throwing away incomplete or long trajectories, the system saves unfinished segments in a replay buffer. In the next training iteration, rollout workers reuse the previous segments and only compute the new, active segment on-policy.

2

Partial Rollout Timeline Simulator

Click "Next Iteration" to see how segments are cached and reused. Notice that only the highlighted Active Segment consumes on-policy compute, while grey segments are loaded instantly from the Replay Buffer.

ITERATION N-1 100% On-Policy Compute

Segment A

Not yet generated

ITERATION N (Current) Reusing Segment A → Generating Segment B

Segment A (Cached)

Segment B

Not yet generated

ITERATION N+1 Awaiting Iteration N completion

Segment A (Cached)

Segment B (Cached)

Segment C

Compute Savings: By reusing Segment A and B from the Replay Buffer, the system avoids generating thousands of context tokens repeatedly. This reduces the computational overhead of long-context RL by up to 60%.

3. Long2Short: Context Compression

While long-CoT models perform exceptionally well, they are slow and expensive to run in production because they consume thousands of tokens per query. Kimi k1.5 introduces several long2short techniques to distill the thinking priors of long-CoT models into highly efficient short-CoT models:

Model Merging: A simple, training-free technique that averages the weights of a long-CoT model and a short-CoT model to boost token efficiency.
Shortest Rejection Sampling: Sampling the same prompt $N=8$ times and picking the shortest correct response for subsequent fine-tuning.
DPO (Direct Preference Optimization): Training the model to prefer shorter, correct answers over long, verbose answers.

3

Long2Short Trade-off Explorer

Hover or tap on the different models to see how the long2short methods manage to compress token usage while maintaining high accuracy on MATH-500.

BASELINE Long-CoT

k1.5-long

Accuracy: 96.2%

Avg Tokens: ~1,400

BEST EFFICIENCY long2short RL

k1.5-short w/ rl

Accuracy: 94.6%

Avg Tokens: ~1,100

ULTRA FAST k1.5-shortest

k1.5-shortest

Accuracy: 88.2%

Avg Tokens: ~650

Visualizing the Token-Accuracy Frontier

Results & Benchmarks

Kimi k1.5 demonstrates state-of-the-art results across math, code, and vision benchmarks, matching OpenAI's o1 on key reasoning metrics.

Limitations & Open Questions

Despite matching top-tier models, the Kimi team highlights several open challenges:

Overthinking / Token Inefficiency: Under pure RL, models tend to generate excessively lengthy reasoning loops. While the length penalty helps, finding the optimal balance between exploration and conciseness remains difficult.
Reward Hacking: Even with strict filters, models sometimes discover superficial patterns to satisfy the verifiers without truly solving the underlying problem.
Advanced Credit Assignment: Because they skip value functions, the model relies purely on final outcome signals. Finding a robust way to perform step-by-step credit assignment without introducing fragile value networks is a key open question.

Glossary

Reinforcement Learning (RL)

A machine learning paradigm where an agent learns to make decisions by performing actions and receiving rewards or penalties. In the context of LLMs, the "actions" are generated tokens, and the "rewards" are based on correctness of the final answer.

Process Reward Models (PRMs)

Reward models that evaluate and score each individual step of a reasoning chain, rather than just the final outcome. Kimi k1.5 explicitly avoids using PRMs, relying instead on outcome-based rewards and scaled context to achieve self-correction.

Chain of Thought (CoT)

An intermediate step-by-step reasoning process generated before producing the final answer. Long-CoT models can generate thousands of tokens planning, reflecting, and backtracking.

Citation

@article{kimi_k1_5_2025,
  title={Kimi k1.5: Scaling Reinforcement Learning with LLMs},
  author={Kimi Team},
  journal={arXiv preprint arXiv:2501.12599v4},
  year={2025}
}