Textbooks Are All You Need

Why This Matters

For years, the dominant paradigm in Large Language Models (LLMs) has been "bigger is better." Modern scaling laws suggest that to improve performance, one must predictably scale up compute, parameter size, and token volume in tandem. This has led to massive, environmentally taxing models trained on trillions of noisy web-scraped tokens.

But what if we are feeding our models the digital equivalent of junk food? Raw web code is often full of boilerplate, poorly documented logic, and non-self-contained snippets. This paper explores a different axis: data quality. By feeding a small model a highly curated "clean diet" of textbook-style explanations and exercises, they demonstrate that we can achieve state-of-the-art reasoning with a fraction of the size.

The Big Idea

The core hypothesis of the paper is that educational value matters more than raw volume. To prove this, the researchers curated a dataset specifically designed to teach basic programming concepts, logic, and algorithmic reasoning. They call this "textbook-quality" data.

Instead of training on raw repositories, they pretrain on a mixture of filtered web code and synthetically generated textbooks, then finetune on a small set of synthetic exercises. This curated diet acts as a catalyst, allowing a relatively small transformer model to construct robust mental representations of code logic.

How It Works

The training pipeline of phi-1 consists of two distinct phases: Pretraining on textbook-quality content, followed by Finetuning on targeted exercises.

1. Curating the Dataset

The dataset consists of less than 7 Billion tokens in total, divided into three main pillars:

def dist(x, y):
    # Computes pairwise L2 distance
    return torch.sqrt(torch.pow(x - y, 2).sum())

self._vim = vim
self._cursor = 0
self._entire_len = 0
self._result = []

2. Architecture & Training Details

The architecture is a classic decoder-only transformer with a few modern optimizations:

• Parallel Configuration: MHA (Multi-Head Attention) and MLP layers are processed in parallel (similar to GPT-J/PaLM) to speed up training throughput.
• FlashAttention: Used for faster context window processing.
• Rotary Position Embeddings (RoPE): For superior sequence-length generalization.

The 1.3B parameters model has 24 layers, a hidden dimension of 2048, and 32 attention heads. It was trained on 8 Nvidia-A100 GPUs using DeepSpeed for only 4 days.

Interactive Demo: Base vs. Fine-Tuned

One of the most remarkable findings in the paper is that fine-tuning on simple exercises unlocked emergent capabilities. The base model (phi-1-base) struggled with complex instructions, but after fine-tuning on CodeExercises, the model (phi-1) suddenly excelled at using external libraries (like PyGame) and complex spatial reasoning, even though those libraries were never in the fine-tuning data.

# Loading...

# Loading...

Results & Benchmarks

Despite having only 1.3 Billion parameters and being trained on 100x fewer tokens than competitors, phi-1 achieves remarkable accuracy on standard Python coding benchmarks.

Addressing Data Contamination

A common criticism of synthetic data models is that they may have "memorized" the evaluation benchmarks during training. To address this, the researchers performed an aggressive decontamination study:

They calculated similarity between the CodeExercises dataset and HumanEval using both N-gram overlap and syntax-based AST (Abstract Syntax Tree) distances. They then pruned up to 40% of the training dataset (removing any files even vaguely similar to HumanEval) and retrained the model.

Limitations & Open Questions

While phi-1 is a breakthrough in data efficiency, the authors outline several key limitations:

• Specialization: The model is highly specialized in Python. It lacks the multi-language versatility of larger models.
• API Knowledge: It lacks domain-specific knowledge of less common packages and APIs, as it was trained primarily on basic concepts.
• Prompt Sensitivity: It is highly sensitive to prompt variations and grammatical errors in instructions, often degrading significantly if the prompt structure changes slightly.