1. Why This Matters
Historically, building more powerful Large Language Models (LLMs) has meant scaling up parameter size. This race led to proprietary giants like GPT-3 (175B) and PaLM (540B). However, the Chinchilla scaling laws suggested that for a given compute budget, a smaller model trained on more data is optimal.
But those scaling laws only consider the training budget. When serving models to millions of users, the inference budget becomes the dominant cost. A massive model like GPT-3 requires expensive multi-GPU clusters to run. LLaMA shifts the paradigm: train smaller models on far more tokens than "optimal" so they are incredibly cheap and easy to host, democratizing access to state-of-the-art AI on everyday developer hardware.
2. The Big Idea
"The preferred model is not the fastest to train, but the fastest at inference... a smaller model trained longer will ultimately be cheaper at inference."
By prioritizing inference efficiency, LLaMA models are trained on up to 1.4 Trillion tokens. To ensure the work could be safely shared and reproduced, the authors restricted themselves entirely to publicly available datasets, breaking the industry trend of relying on proprietary, undocumented training corpora.
3. Pre-training Data Mixture
LLaMA is trained on a carefully curated mixture of public datasets. Each source went through robust preprocessing pipelines to ensure high-quality text representation. Explore the interactive visual below to see how the dataset is structured and what filtering rules were applied.
Interactive Dataset Explorer
Click on any dataset slice below to reveal its specific preprocessing details and disk footprint.
English CommonCrawl
Preprocessed with the CCNet pipeline. Deduplicated at the line level, language filtered using a fastText classifier, and low-quality content removed with an n-gram language model.
4. Architectural Upgrades
LLaMA is based on the standard Transformer architecture, but introduces three critical modifications inspired by other successful architectures. These changes significantly improve training stability and overall performance.
Transformer Block Comparison
Pre-normalization (RMSNorm)
To improve training stability, LLaMA normalizes the input of each transformer sub-layer instead of the output. It uses RMSNorm, which is computationally cheaper than standard LayerNorm as it ignores mean-centering.
5. The Inference Compute Crossover
The core thesis of LLaMA is that extra training compute pays off during inference. If you train a smaller model for longer, you spend more money upfront on training, but you save money on every single inference query.
This interactive calculator visualizes this exact crossover point. Compare LLaMA-13B (trained on 1.0T tokens) against GPT-3 (175B) (trained on 300B tokens) to see how soon the smaller model becomes more compute-efficient overall.
Compute Crossover Calculator
The number of days of continuous operation at this scale before LLaMA-13B's training investment saves more aggregate FLOPs than using a GPT-3 175B class model.
6. Performance & Results
Despite its smaller size, LLaMA achieves remarkable performance. Specifically, LLaMA-13B outperforms GPT-3 on most standard commonsense reasoning benchmarks, while being 10x smaller, allowing it to run on a single developer GPU.
| Model | Parameters | MMLU (5-shot) | HumanEval (pass@1) |
|---|---|---|---|
| GPT-3 | 175B | 43.9% | - |
| LLaMA-13B | 13B | 46.9% | 15.8% |
| Chinchilla | 70B | 67.5% | - |
| LLaMA-65B | 65B | 63.4% | 23.7% |
7. Limitations & Biases
Like most large language models trained on web data, LLaMA carries significant risks of generating toxic language, propagating societal biases, and hallucinating false facts.
The authors evaluated LLaMA-65B on benchmarks like CrowS-Pairs (measuring bias across categories like gender, religion, and race) and RealToxicityPrompts. They note that toxicity scores generally increase with model size, highlighting the urgent need for instruction tuning and alignment strategies (such as RLHF) before deploying these models in user-facing applications.
8. Glossary
Chinchilla Scaling Laws
Empirical scaling laws introduced by DeepMind showing that for optimal training compute efficiency, model size and training dataset size should scale in equal proportion.
RMSNorm (Root Mean Square Normalization)
A variant of Layer Normalization that scales inputs by the root mean square instead of both mean and variance, saving computational overhead while preserving training stability.
SwiGLU Activation Function
An activation function combining Swish and Gated Linear Units. It replaces standard ReLU/GELU in the feed-forward network, leading to better convergence.
Rotary Position Embeddings (RoPE)
A method of incorporating positional information by rotating the Query and Key vectors in the complex plane, allowing better extrapolation to longer context lengths.
Cite This Work
@article{touvron2023llama,
title={LLaMA: Open and Efficient Foundation Language Models},
author={Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timothee and Rozi{\`e}re, Baptiste and Goyal, Naman and Hambro, Eric and Azhar, Faisal and others},
journal={arXiv preprint arXiv:2302.13971},
year={2023}
}