Llama 2: Open Foundation and Fine-Tuned Chat Models
Authors: Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, and the Meta GenAI Team.
TL;DR
Meta introduces Llama 2, a suite of open-weights LLMs (7B, 13B, and 70B parameters) optimized for dialogue. By utilizing 2 trillion pretraining tokens, doubling context lengths to 4096, and implementing iterative RLHF alongside a novel "Ghost Attention" mechanism, Llama 2-Chat matches or exceeds commercial closed-source baselines on standard safety and helpfulness evaluations.
Why This Matters
Before Llama 2, the AI landscape was starkly divided. On one side stood highly capable, proprietary models like ChatGPT and Claude, locked behind APIs and proprietary walls. On the other side were open-source alternatives that, while promising, struggled with multi-turn dialogue consistency, factual alignment, and robust safety guardrails.
Llama 2 bridged this gap by releasing open-weights models with a commercial-friendly license. It demonstrated that meticulous data curation, iterative alignment via Reinforcement Learning from Human Feedback (RLHF)A method that uses human feedback to train a reward model, which then guides the LLM to generate preferred outputs., and targeted architectural tweaks could produce an open-source model capable of competing head-to-head with closed-source giants.
The Big Idea
"The capabilities of LLMs are remarkable considering the seemingly straightforward nature of the training methodology... Alignment, however, is what transforms these raw statistical predictors into helpful, safe assistants."
The core contribution of Llama 2 lies not in a brand-new neural architecture, but in its rigorous recipe for alignment. By training separate reward models for *helpfulness* and *safety*, employing iterative *Rejection Sampling* combined with *Proximal Policy Optimization (PPO)*, and introducing *Ghost Attention* to force systemic multi-turn instruction following, Llama 2 establishes a highly reproducible blueprint for creating safe, state-of-the-art conversational agents.
1. Pretraining & Grouped-Query Attention (GQA)
Llama 2 increases pretraining scale significantly, training on 2.0 Trillion tokens (a 40% increase over Llama 1) and doubling the context window to 4096 tokens.
To prevent the memory footprint of the KV CacheA key-value storage mechanism used during autoregressive decoding to avoid recalculating attention vectors for previous tokens. from exploding at larger batch sizes and context lengths, the 34B and 70B parameter models adopt Grouped-Query Attention (GQA). Instead of giving every query head its own key/value head (Multi-Head Attention) or forcing all query heads to share a single key/value head (Multi-Query Attention), GQA groups query heads to share a smaller subset of key/value heads.
Interactive GQA & KV Cache Calculator
Toggle between attention mechanisms and adjust the sliders to see how Grouped-Query Attention drastically reduces KV Cache memory usage during inference.
2. Supervised Fine-Tuning & Iterative RLHF
Llama 2-Chat's training starts with Supervised Fine-Tuning (SFT)The process of training a pretrained language model on high-quality, human-curated prompt-response pairs to establish a baseline of helpful behavior.. Crucially, the authors discovered that "Quality is all you need": a modest set of 27,540 highly clean, vendor-annotated prompt-response pairs yielded far better results than millions of noisier, web-scraped examples.
Following SFT, the model undergoes iterative RLHF. Instead of training a single reward model, Llama 2 trains two separate reward models: one optimized for *Helpfulness* and another for *Safety*. This prevents the model from becoming overly conservative (refusing harmless prompts out of extreme safety caution) or overly compliant (answering dangerous prompts).
The alignment loop alternates between Rejection Sampling (sampling $K$ candidate outputs from the model and selecting the best one according to the reward models to update gradients) and Proximal Policy Optimization (PPO).
Reward Score Distribution Shift
Drag the slider to see how iterative RLHF (from SFT to RLHF V5) systematically eliminates the "bad tail" of low-scoring generations, shifting the model's output distribution toward high-quality responses.
3. Multi-Turn Consistency & Ghost Attention (GAtt)
Standard SFT and RLHF models often suffer from "instruction decay" during long conversations. If you ask a model to "Respond only in Haiku," it might obey for the first turn, but forget the constraint by turn three or four.
To solve this, Llama 2 introduces Ghost Attention (GAtt). GAtt synthetically modifies multi-turn training data. It appends the system instruction $inst$ to all user messages in a conversation, but sets the loss to $0$ for all previous turns during gradient descent. This forces the model's self-attention layers to maintain a strong, active link to the initial system prompt across the entire context window.
Ghost Attention (GAtt) Live Simulator
Toggle between a standard model and a GAtt-aligned model, then click through the multi-turn dialogue to see how standard models forget constraints while GAtt maintains strong attention to system instructions.
"Constraint: Respond ONLY in Haiku"
Results & Benchmarks
Llama 2 models were evaluated extensively against both open-source and closed-source baselines. On academic benchmarks like MMLU (academic knowledge) and GSM8K (math reasoning), Llama 2 70B significantly outperformed all other open-source alternatives and closed-source models like PaLM (540B) on several tasks.
Limitations & Open Questions
Despite its impressive performance, Llama 2 exhibits several limitations common to modern autoregressive transformers:
- Language Bias: The pretraining corpus is 89.7% English, meaning performance in other languages remains fragile and should be used with caution.
- Knowledge Cutoff: The training data has a cutoff of September 2022, making the model unaware of more recent events.
- Over-Conservatism: Due to strict safety alignment, early versions of Llama 2-Chat sometimes "over-refuse" benign prompts that contain sensitive keywords (e.g., refusing to explain "how to kill a process" in Linux).
Glossary
Grouped-Query Attention (GQA)
An attention mechanism that groups query heads together to share a single key and value head. This serves as an intermediate strategy between Multi-Head Attention (MHA) and Multi-Query Attention (MQA), balancing speed and model capacity.
Proximal Policy Optimization (PPO)
A reinforcement learning algorithm used to optimize the model policy during alignment. It limits policy updates to a safe boundary, preventing the model from collapsing or diverging during training.
Rejection Sampling
A technique where multiple candidate responses are generated by the LLM, scored by a reward model, and only the highest-scoring response is selected to update the model parameters.
Cite This Work
@article{touvron2023llama,
title={Llama 2: Open Foundation and Fine-Tuned Chat Models},
author={Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others},
journal={arXiv preprint arXiv:2307.09288},
year={2023}
}