Qwen3 Technical Report: Unified Reasoning & Dynamic Thinking Budgets
By Qwen Team • May 15, 2025
TL;DR: Qwen3 introduces a unified framework that seamlessly integrates rapid, context-driven response generation with deep, multi-step reasoning (thinking mode) in a single model. By introducing an adaptive "thinking budget" and highly efficient strong-to-weak distillation, Qwen3 matches proprietary reasoning models at a fraction of the inference and training cost.
Why This Matters
Historically, AI developers and users had to choose between two completely different classes of Large Language Models (LLMs): chat-optimized models (like GPT-4o or Qwen2.5) that deliver rapid, conversational replies, and dedicated reasoning models (like QwQ or DeepSeek-R1) that spend significant computational time generating a long, internal Chain-of-Thought (CoT).
This division forced developers to implement complex routing logic, switch between separate API endpoints, and pay high latency penalties even for trivial queries. Qwen3 breaks down this wall by integrating both behaviors into a single model, allowing users to dynamically dial the reasoning depth up or down on the fly.
The Big Idea
The core contribution of Qwen3 is the Thinking Mode Fusion framework. Instead of treating reasoning as a completely separate post-training regime, Qwen3 models are trained to natively understand special instruction flags (such as /think and /no_think) in their chat templates.
When thinking is enabled, the model outputs its logical trace inside structured <think>...</think> blocks. Crucially, the model is trained to handle incomplete thinking gracefully. If the user constraints limit the generation mid-reasoning (using a token budget), Qwen3 can instantly pivot, close the thinking block, and output a coherent final answer based on its partial reasoning.
Interactive Playground
Experience Qwen3's dual operating modes and dynamic thinking budget control below.
1 Dynamic Mode Switcher
Toggle between instant conversational mode and deep reasoning mode.
2 Thinking Budget & Performance Scaling
Adjust the thinking token budget to simulate how Qwen3-235B scales accuracy on complex benchmarks.
Budget Impact Analysis
At 8K tokens, Qwen3 balances inference speed with deep step-by-step reasoning, capturing 71.0% on GPQA Diamond and 85.1% on AIME'24.
How It Works
The architectural foundation and multi-stage pre-training recipe.
1. Dual-Architecture Scaling
Qwen3 scales from ultra-portable 0.6B edge models up to the flagship 235B Mixture-of-Experts (MoE) model. The flagship features a total of 235 billion parameters, but dynamically activates only 22 billion parameters per token.
Key architectural features include Grouped Query Attention (GQA), SwiGLU activations, QK-Norm to stabilize training, and the removal of standard QKV biases.
2. Three-Stage Pre-training
Qwen3 models are trained on a massive 36 trillion tokens corpus across 119 languages. The pre-training is divided into three highly specialized stages:
- S1: General Stage (30T tokens): Instills general world knowledge and broad language proficiency.
- S2: Reasoning Stage (5T tokens): Upweights STEM, coding, and synthetic mathematical data.
- S3: Long Context Stage (hundreds of billions): Extends context length from 4,096 to 32,768 tokens using YARN and Dual Chunk Attention.
Post-Training & Distillation
How flagship and lightweight models achieve reasoning without massive compute.
Post-Training Pipeline Explorer
Click on the pipeline stages to explore the data curation, objectives, and training methods.
Stage 1: Long-CoT Cold Start
Curates a high-quality dataset across math, code, and logic. Uses Qwen2.5-72B-Instruct to filter out simple queries and QwQ-32B to generate reasoning steps. Minimizing training samples and steps here instills foundational reasoning patterns without limiting future exploration during reinforcement learning.
Empirical Results
Qwen3 compared against top open-source and proprietary architectures.
Flagship Qwen3-235B outperforms DeepSeek-R1 with only 60% of the activated parameters.
Demonstrates state-of-the-art code generation and execution capabilities.
Reaches the skill levels of advanced human competitive programmers.
Limitations & Open Questions
Despite outstanding reasoning performance, the Qwen Team highlights a critical trade-off: Thinking Mode Fusion and General RL do not improve performance on all tasks.
Specifically, for highly challenging tasks like AIME'24 and LiveCodeBench, the model's peak reasoning accuracy in thinking mode actually decreased slightly after the final stages of training. The authors conjecture that training the model on a broader range of general tasks compromises its highly specialized capabilities on complex math and coding.
Glossary
GRPO (Group Relative Policy Optimization)
An efficient reinforcement learning algorithm that updates model parameters by comparing a group of outputs generated from the same prompt, avoiding the need for a separate critic model.
Thinking Mode Fusion
The post-training process that blends reasoning paths (CoT) and direct response datasets, enabling a single model to support both modes via chat templates.
Strong-to-Weak Distillation
A method where logits from a powerful teacher model (e.g., Qwen3-235B) are transferred to a smaller student model, minimizing KL divergence to instill reasoning at 1/10th of the training cost.
Citation
@article{qwen3techreport2025,
title={Qwen3 Technical Report},
author={Qwen Team},
journal={arXiv preprint arXiv:2505.09388},
year={2025}
}