Instruction Tuning — FlashPapers Concept

Alignment Supervised Fine-Tuning (SFT)

Instruction Tuning

The process of fine-tuning a pretrained language model on a dataset of task descriptions (instructions) and corresponding responses to align its behavior with user intent.

In One Sentence

Instruction tuning bridges the gap between next-token prediction on raw web text and helpful assistant behavior by fine-tuning models on structured prompt-response pairs, masking loss on the prompt itself.

Intuition

Base language models are trained to be master copy-pastes of the internet. If you give a raw base model the prompt "Write a poem about a tree.", it might not write a poem. Instead, it might output "Write a poem about a flower. Write a poem about a rock." because it treats your input as the start of a list of writing prompts.

Instruction Tuning changes this behavior. By showing the model thousands of examples where an instruction is followed immediately by its correct completion, the model learns to shift its operational mode from document completion to instruction execution.

Crucially, during this training process, we only calculate gradients and update weights based on the model's ability to predict the response, not the instruction. This prevents the model from wasting capacity on memorizing how prompts are phrased.

Formal Definition

Let $x = (x_1, \dots, x_U)$ be the sequence of tokens representing the instruction (prompt), and $y = (y_1, \dots, y_V)$ be the sequence of tokens representing the target response.

The sequence is concatenated as $W = [x; y]$. During training, we optimize the model parameters $\theta$ to maximize the conditional log-likelihood of the response tokens $y$ given the prompt $x$:

$$\mathcal{L}(\theta) = \sum_{t=1}^{V} \log P_\theta(y_t \mid x, y_{

In practice, this is implemented by setting the loss weights (or labels) of all tokens in $x$ to a special ignore index (typically -100 in PyTorch), ensuring that gradients are only backpropagated through the response tokens $y$.

Interactive Walkthrough: Loss Masking

Toggle Training Objective

See how the loss calculation shifts between pre-training and instruction tuning (SFT).

Token Sequence & Loss Weights

Token 1 User: Loss: 1.0

Token 2 Write Loss: 1.0

Token 3 a Loss: 1.0

Token 4 poem. Loss: 1.0

Token 5 \n\n Loss: 1.0

Token 6 In Loss: 1.0

Token 7 silicon Loss: 1.0

Token 8 dreams... Loss: 1.0

In Standard Pre-training, the model calculates loss on every single token in the sequence. It is penalized for failing to predict the prompt "User: Write a poem." just as much as the response.

Trade-offs & Usage

When to Use

• You want to convert a raw base model into an interactive chat assistant.
• You need to enforce specific output formats (e.g., JSON, XML, or Chain-of-Thought reasoning).
• You want to enable zero-shot generalization to tasks the model has never explicitly seen.

When to Avoid

• To inject core factual knowledge: Pre-training or continual pre-training on raw corpora is far more efficient for learning new domains.
• When few-shot in-context learning (prompting) already achieves the desired accuracy without training overhead.

History & Evolution

March 2022

InstructGPT (OpenAI)

OpenAI demonstrates that fine-tuning GPT-3 on human-written demonstrations (SFT) and subsequent RLHF dramatically improves user preference, enabling a 1.3B parameter model to outperform a 175B base model on instruction following (Ouyang et al., arXiv:2203.02155).

October 2022

Scaling Instruction-Finetuned Models (Flan)

Google researchers scale instruction tuning to 1.8K tasks with Flan-PaLM and Flan-T5, showing that multi-task instruction tuning dramatically improves zero-shot generalization and chain-of-thought capabilities (Chung et al., arXiv:2210.11416).

December 2022

Self-Instruct

Wang et al. introduce a framework to bootstrap instruction-following datasets using a model's own generations, reducing reliance on expensive human annotations (Wang et al., arXiv:2212.10560).

November 2024

Tulu 3 (Allen Institute for AI)

AI2 releases Tulu 3, providing fully open state-of-the-art post-training recipes combining SFT, DPO, and RLVR to match or exceed proprietary models (arXiv:2411.15124).

Implementation (PyTorch-style)

sft_loss.py Active

# PyTorch-like loss calculation with prompt masking
import torch
import torch.nn as nn

def compute_sft_loss(model, input_ids, attention_mask, labels, loss_fn):
    """
    input_ids: Concatenated prompt + response tokens [Batch, SeqLen]
    labels: Same shape as input_ids, but prompt tokens are replaced with -100
    """
    outputs = model(input_ids=input_ids, attention_mask=attention_mask)
    logits = outputs.logits
    
    # Shift logits and labels for standard autoregressive next-token prediction
    shift_logits = logits[..., :-1, :].contiguous()
    shift_labels = labels[..., 1:].contiguous()
    
    # Flatten and compute loss. PyTorch's CrossEntropyLoss ignores -100 by default.
    loss = loss_fn(
        shift_logits.view(-1, shift_logits.size(-1)), 
        shift_labels.view(-1)
    )
    return loss

Common Variants

Multi-Task SFT (Flan)

Formatting standard NLP benchmarks (classification, summarization) as instructions to maximize generalization to unseen prompts.

Self-Instruct (Synthetic SFT)

Using a high-quality teacher model to generate both the instructions and responses, bypassing human annotation bottlenecks.

Parameter-Efficient SFT

Applying adapter methods like LoRA to freeze the base model and only train a small percentage of parameters on instruction sets.

Representative Flash Papers

ryhchkpe82

Language Models are Few-Shot Learners

Introduces GPT-3, demonstrating strong zero/few-shot performance, which served as the baseline model for the original InstructGPT alignment work.

pwtp72pyqh

Exploring the Limits of Transfer Learning with T5

Establishes the unified text-to-text framework, which laid the architectural and task-formatting foundations for Flan-T5 instruction tuning.

Grounding References

[1] Ouyang et al. "Training language models to follow instructions with human feedback." arXiv:2203.02155 (InstructGPT)
[2] Chung et al. "Scaling Instruction-Finetuned Language Models." arXiv:2210.11416 (Flan-T5/Flan-PaLM)
[3] Wang et al. "Self-Instruct: Aligning Language Models with Self-Generated Instructions." arXiv:2212.10560
[4] Allen Institute for AI. "Tulu 3: Pushing Frontiers in Open Language Model Post-Training." arXiv:2411.15124

Made with Flash Papers — make your own ↗