Self-Instruct: Aligning LMs with Self-Generated Instructions

Why This Matters

Large Language Models (LLMs) are incredibly powerful, but their raw pretraining state makes them hard to control. To make them useful assistants, we use instruction tuning—training them to respond to natural language prompts.

However, creating high-quality instruction datasets is extremely expensive and bottlenecked by human creativity. Humans tend to write instructions for a narrow band of popular tasks (like simple Q&A or summarization), leaving a massive tail of potential applications unexplored.

The Core Problem: Human annotation is slow, expensive, and lacks the diverse scale required to fully cover the vast space of possible tasks that models are capable of performing.

The Big Idea

What if we let the language model align itself? Pretrained LMs already possess latent knowledge of thousands of tasks. SELF-INSTRUCT bootstraps this latent knowledge by prompting the model to generate its own instructions, inputs, and outputs, filtering out low-quality generations, and then finetuning itself on this self-generated instruction set.

How It Works

The framework operates in a continuous loop, starting from a tiny seed pool of 175 human-written tasks. The process is divided into four main steps:

1

Instruction Generation

The model is prompted with 8 tasks (6 human-written, 2 previously model-generated) to brainstorm new instructions for novel tasks.

2

Classification Task Identification

The pipeline classifies if the generated instruction is a classification task (requiring finite label options) or a generation task.

3

Instance Generation

Depending on the task type, the model generates input/output pairs using either an Input-First or Output-First approach to avoid label bias.

4

Filtering & Postprocessing

To maintain quality and diversity, tasks are filtered out if they are too similar to existing tasks (using ROUGE-L similarity) or fail basic heuristics.

Widget 1: Pipeline Simulator

Step-by-step Interactive Demo

Select a task style below to see how the Self-Instruct pipeline processes it from raw generation to final pool inclusion.

Step 1: Generated Instruction

"Write a letter from the perspective of a cat."

Step 2: Classification Identification

No (Generation Task)

Step 3: Instance Generation (Input-First vs Output-First)

Input: Null

Output: "Dear [Owner], I am writing to you today because..."

Step 4: Filtering & Similarity Check

PASSED ROUGE-L similarity with pool is 0.18 (< 0.70)

Widget 2: ROUGE-L Similarity Filter

Diversity Heuristics

To maintain task diversity, new instructions are rejected if their ROUGE-L similarity with any existing task in the pool is above a threshold. Use the slider to adjust the threshold and see what gets filtered.

ROUGE-L Threshold: 0.70 Strict (0.3) → Loose (1.0)

"Convert Fahrenheit to Celsius"

Max similarity with seed pool: 0.21

"Translate English to French"

Max similarity with seed pool: 0.65

"Write a short essay on school safety"

Max similarity with seed pool: 0.85 (Very close to "Write an essay on school safety")

Experimental Results

The authors evaluated models fine-tuned with Self-Instruct against public instruction-tuned datasets and OpenAI's proprietary InstructGPT models.

On the SUPER-NATURALINSTRUCTIONS (SuperNI) benchmark, GPT3_SELF-INST achieved a 39.9 ROUGE-L score, representing a 33.1% absolute improvement over vanilla GPT3 (6.8), and nearly matching InstructGPT₀₀₁ (40.8).

Widget 3: Human Evaluation on Novel Tasks

User-oriented Evaluation (252 Tasks)

Human evaluators rated model responses on a 4-level scale (A: Perfect, B: Minor errors, C: Major errors, D: Completely invalid). Select a model to inspect its quality distribution.

Rating A (Perfect)

44%

Rating B (Minor)

19%

Rating C (Major)

26%

Rating D (Failed)

11%

Limitations & Biases

While highly effective, the authors identify several critical limitations with bootstrapping alignment entirely from language models:

Tail Phenomena: Models struggle with uncommon, highly creative, or niche instructions that fall outside the mainstream pretraining distribution.
Reinforcing Biases: The iterative algorithm can amplify pre-existing social biases (gender, race, etc.) or produce severely unbalanced label spaces.
Dependence on Large Models: The success of the method scales with the base model's size. Smaller models struggle to bootstrap meaningful task diversity.

Glossary

Instruction Tuning

The process of finetuning a pretrained language model on a dataset of task instructions and target completions, enabling it to generalize to new, unseen instructions at inference time.

Bootstrapping

A self-sustaining process where a system uses its own output or a small initial state (the seed pool) to generate larger, more complex inputs to train or improve itself.

ROUGE-L

Recall-Oriented Understudy for Gisting Evaluation (using Longest Common Subsequence). It measures the overlap of word sequences between two texts, used here to prevent the generation of redundant tasks.

Citation

@inproceedings{wang2023selfinstruct,
  title={Self-Instruct: Aligning Language Models with Self-Generated Instructions},
  author={Wang, Yizhong and Kordi, Yeganeh and Mishra, Swaroop and Liu, Alisa and Smith, Noah A. and Khashabi, Daniel and Hajishirzi, Hannaneh},
  booktitle={Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL)},
  year={2023}
}

SELF-INSTRUCT: Aligning Language Models with Self-Generated Instructions

TL;DR