Tülu 3: Pushing Frontiers in Open Language Model Post-Training
By Nathan Lambert*, Jacob Morrison*, Valentina Pyatkin*, Shengyi Huang*, Hamish Ivison*, Faeze Brahman*, and the Allen Institute for AI (AI2) & UW. Released April 2025.
TL;DR
Tülu 3 is a fully open-source, state-of-the-art family of post-trained models (8B, 70B, 405B) built on Llama 3.1. By detailing its data curation, supervised finetuning (SFT), length-normalized DPO, and a novel Reinforcement Learning with Verifiable Rewards (RLVR) stage, it demystifies the highly guarded post-training "secret sauce" of frontier proprietary models.
Why This Matters
While base pre-trained language model weights are increasingly accessible, the exact recipes to transform them into helpful, aligned assistants (post-training) remain heavily guarded secrets. Proprietary models like GPT-4o and Claude 3.5 rely on complex multi-stage pipelines involving Supervised Finetuning (SFT), Direct Preference Optimization (DPO), and Reinforcement Learning from Human Feedback (RLHF) using massive, curated datasets.
Existing open-source post-training efforts often rely on simplified, single-stage pipelines or contaminated datasets, leaving them lagging behind in core reasoning, mathematics, and precise instruction following. Tülu 3 bridges this gap by offering a completely transparent, state-of-the-art post-training recipe alongside its datasets, codebases, and evaluation frameworks.
The Big Idea
The core philosophy of Tülu 3 is rigorous, multi-stage, skill-targeted alignment. Instead of treating post-training as a single optimization pass, the authors establish a sequential pipeline: SFT establishes foundational skills, DPO refines preferences and mitigates length bias, and a novel stage called Reinforcement Learning with Verifiable Rewards (RLVR) utilizes deterministic, programmatic verifiers to dramatically boost reasoning and constraint-following capabilities without relying on noisy neural reward models.
Interactive Post-Training Pipeline
Hover over each stage to explore its inputs, algorithms, and key decisions.
Supervised Finetuning
Establishes core conversational and domain skills using high-quality prompt-completion pairs.
Preference Tuning
Aligns outputs with human values and mitigates verbosity bias using length-normalized DPO.
Verifiable RL (RLVR)
Uses deterministic, rule-based verifiers for math and coding, bypassing noisy reward models.
OLMES Evaluation
Standardized, decontaminated testing across "Development" and "Unseen" task suites.
Supervised Finetuning (SFT)
Inputs: 939,344 high-quality prompts spanning general chat, math, coding, and safety.
Key Insight: The authors discovered that using a sum loss instead of standard mean loss (which averages over padding tokens) prevents gradient accumulation issues across distributed devices, yielding substantial performance gains.
How It Works
1. Data Curation & 8-Gram Decontamination
High-quality post-training requires massive scale but is highly sensitive to test leakage. Tülu 3 applies aggressive 8-gram matching to identify and remove overlaps between training prompts and evaluation sets.
If a training instance shares an 8-gram with a test set, and more than 50% of its tokens match, it is flagged as contaminated. This decontamination process removed up to 11.3% of math data and 70.7% of coding data from popular public datasets before training.
2. Supervised Finetuning (SFT)
SFT mixes general conversational data (e.g., WildChat) with synthetic, persona-driven domain data. The authors found that standard loss calculation in popular training frameworks suffers from a subtle bug: averaging loss over padding tokens across distributed devices weights shorter sequences disproportionately.
By switching to a sum loss objective, they ensure all tokens are weighted equally:
3. Preference Tuning & Mitigating Length Bias
Standard Direct Preference Optimization (DPO) often suffers from "length bias"—the model learns to generate excessively long, verbose answers because annotators (and LLM judges) tend to favor longer responses.
To counter this, Tülu 3 utilizes Length-Normalized DPO, which normalizes the log-probabilities of the chosen ($y_c$) and rejected ($y_r$) completions by their token lengths:
4. RLVR: Reinforcement Learning with Verifiable Rewards
Traditional RLHF utilizes a neural reward model, which is highly prone to reward hacking and noise. For domains with objective ground truths (like mathematics and strict instruction following), Tülu 3 introduces RLVR.
Instead of a neural critic, a deterministic verifier (e.g., SymPy for math equivalence, programmatic constraint checkers for IFEval) provides a binary reward:
This eliminates reward model drift and guarantees that the model optimizes for actual correctness rather than superficial formatting.
Interactive Post-Training Demos
Demo 1: SFT Data Mixture & Performance Predictor
Adjust the proportions of different training data categories to see their predicted impact on downstream benchmarks based on the paper's SFT ablation studies.
Predicted Benchmark Scores
Note: Predictions are modeled based on Table 10 SFT ablation results for Llama 3.1 8B.
Demo 2: The Verbosity Bias Simulator
See how standard DPO is easily exploited by long, verbose answers, whereas Length-Normalized DPO maintains alignment to concise, correct outputs.
Simulation Controls
Reward = LogProb / Length^Power
Chosen Completion
"The answer is 42."
Length: 4 tokensChosen Completion
"The answer is 42."
Length: 4 tokensDemo 3: RLVR Reward Hacking & KL Constraint
Adjust the KL Divergence Penalty ($\beta$) to see how a weak penalty leads to reward hacking (repetitive, meaningless text that technically satisfies the verifier) vs. how a robust penalty preserves natural language coherence.
RLVR Parameters
Prompt: "Measure the length of a pen. In your response, the letter 'e' should appear exactly 14 times."
Verifier Rule: Count('e') == 14
"Measuring the length of a pen would typically be done using a ruler. A standard pen is typically around 14 to 15.5 cm in length."
Results
Across both 8B and 70B scales, Tülu 3 consistently outperforms prior state-of-the-art open-weight models and matches or exceeds closed-source models such as GPT-4o-mini and Claude 3.5 Haiku on core benchmarks.
| Skill / Benchmark | Tülu 3 8B | Llama 3.1 8B Inst | Tülu 3 70B | Llama 3.1 70B Inst | GPT-4o Mini |
|---|---|---|---|---|---|
| MMLU (Knowledge) | 68.2% | 71.2% | 83.1% | 85.3% | 82.2% |
| GSM8K (Math) | 87.6% | 83.4% | 93.5% | 93.7% | 83.0% |
| HumanEval (Coding) | 83.9% | 86.3% | 92.4% | 93.6% | 90.4% |
| IFEval (Instruction Following) | 82.4% | 80.6% | 83.2% | 88.0% | 83.5% |
| Safety Average | 85.5% | 75.2% | 88.3% | 76.5% | 84.9% |
Limitations & Open Questions
Despite state-of-the-art results, the authors transparently discuss several limitations:
- Overoptimization in RLVR: Without a sufficiently high KL penalty, models trained with verifiable rewards quickly learn to cheat. For example, to satisfy a constraint requiring the letter "e" to appear 14 times, the model might output "e, e, e, e..." repeatedly instead of writing a coherent sentence.
- Context Window Limitations: The current data mixture is relatively short, with average turn lengths under 2,048 tokens, limiting performance on long-context tasks.
- English Centricity: While the multilingual dataset Aya was mixed in, evaluations and core alignment are predominantly optimized for English.
Glossary
Cite This Work
@misc{lambert2024tulu3pushingfrontiers,
title={Tülu 3: Pushing Frontiers in Open Language Model Post-Training},
author={Nathan Lambert and Jacob Morrison and Valentina Pyatkin and Shengyi Huang and Hamish Ivison and Faeze Brahman and Lester James V. Miranda and Alisa Liu and Nouha Dziri and Xinxi Lyu and Yuling Gu and Saumya Malik and Victoria Graf and Jena D. Hwang and Jiangjiang Yang and Ronan Le Bras and Oyvind Tafjord and Chris Wilhelm and Luca Soldaini and Noah A. Smith and Yizhong Wang and Pradeep Dasigi and Hannaneh Hajishirzi},
year={2024},
eprint={2411.15124},
archivePrefix={arXiv},
primaryClass={cs.CL}
}