BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
By Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova · Google AI Language
TL;DR
BERT is a breakthrough language representation model that pre-trains deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context. By using a novel Masked Language Model (MLM) objective, BERT allows a single pre-trained architecture to be fine-tuned with just one additional output layer to achieve state-of-the-art results on 11 diverse NLP tasks.
Why This Matters
Before BERT, language model pre-training was constrained by a fundamental limitation: unidirectionality. Traditional language models (like OpenAI's GPT-1) were trained to predict the next word in a sequence from left to right. While this works beautifully for generating text, it is highly sub-optimal for understanding sentence-level or token-level semantics.
Consider the word "bank" in these two sentences:
- "He walked along the river bank."
- "He deposited money in the bank."
A left-to-right model processing the word "bank" only has access to "He walked along the river" or "He deposited money in the". It cannot look ahead to see "river" or "money" if those clues happen to appear later in the sentence. While models like ELMo tried to solve this by concatenating a left-to-right and a right-to-left model, this approach is shallow and does not allow the representations to interact across layers.
The Big Idea
BERT overcomes the unidirectionality constraint by using a "Masked Language Model" (MLM) pre-training objective. Instead of predicting the next token, BERT randomly masks a percentage of the input tokens and attempts to predict those masked words based on both their left and right context. This allows the model to build a deep, bidirectional representation at every layer of the Transformer encoder.
1. Masked LM (MLM)
Masks 15% of input tokens. Forces the model to fuse left and right contexts to reconstruct the corrupted words.
2. Next Sentence Prediction (NSP)
Predicts whether Sentence B actually follows Sentence A. Crucial for understanding relationship-heavy tasks like QA and NLI.
How It Works
BERT uses a multi-layer bidirectional Transformer encoder. The paper defines two primary model sizes:
- $\text{BERT}_{\text{BASE}}$: $L=12$ layers, $H=768$ hidden size, $A=12$ attention heads (110M parameters).
- $\text{BERT}_{\text{LARGE}}$: $L=24$ layers, $H=1024$ hidden size, $A=16$ attention heads (340M parameters).
Widget 1 Interactive Attention Matrix Explorer
Select an architecture and hover over tokens to see how context flows during pre-training.
BERT: Tokens attend to both left and right context in all layers.
The MLM Masking Strategy (80-10-10 Rule)
To prevent a mismatch between pre-training and fine-tuning (since the [MASK] token never appears during fine-tuning), the authors devised a clever masking strategy. For the 15% of tokens chosen to be masked:
- 80% of the time: Replace the word with the actual [MASK] token.
- 10% of the time: Replace the word with a random token.
- 10% of the time: Keep the word unchanged (biasing the model towards the actual observed word).
Widget 2 Stochastic MLM Masking Simulator
Simulate how BERT corrupts training text. Click "Run Simulation" to stochastically apply the 80/10/10 rule.
Widget 3 Transformer Parameter & Dimension Calculator
Adjust the sliders to build a custom Transformer Encoder and calculate its exact mathematical parameter footprint.
Total stacked Transformer Encoder blocks.
Dimensionality of the encoder layers and pooler representations.
Number of parallel self-attention subspaces.
Empirical Results
BERT achieved state-of-the-art results on 11 natural language processing tasks at the time of publication, including GLUE, SQuAD v1.1, and SQuAD v2.0. Below is a comparison of BERT against contemporary baselines.
Benchmark Performance Comparison
Select a dataset to view the performance comparison.
Limitations & Open Questions
While BERT fundamentally changed natural language processing, the authors and subsequent researchers identified several limitations:
- Pre-train / Fine-tune Mismatch: Although the 80/10/10 masking strategy mitigates this, the [MASK] token never appears during fine-tuning, leaving a slight discrepancy between the training distributions.
- Computational Overhead: Pre-training BERT is extremely expensive. $\text{BERT}_{\text{LARGE}}$ required 64 Cloud TPUs for 4 days, making it inaccessible for small labs to train from scratch.
- Quadratic Complexity: Like all standard Transformers, BERT's self-attention mechanism scales quadratically $O(N^2)$ with sequence length, making long-document processing difficult.
- Next Sentence Prediction (NSP) Utility: Later ablation studies (such as RoBERTa) showed that removing the NSP task sometimes improves or maintains performance on downstream tasks if the input sequences are formatted differently.
Glossary
Bidirectional Encoder
An architecture that processes input sequences by looking at both the left (past) and right (future) context simultaneously at every layer, as opposed to causal/unidirectional encoders which only process tokens left-to-right.
WordPiece Embeddings
A sub-word tokenization system that breaks rare words into smaller units (e.g., "playing" becomes "play" and "##ing") to handle out-of-vocabulary words gracefully while maintaining a compact vocabulary size (usually 30,000 for BERT).
[CLS] and [SEP] Tokens
Special tokens added to inputs. [CLS] (classification) is placed at the start of every sequence; its final hidden state is used for classification tasks. [SEP] (separator) is used to split two distinct sentences in tasks like question answering.
Cite this Paper
@article{devlin2018bert,
title={Bert: Pre-training of deep bidirectional transformers for language understanding},
author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
journal={arXiv preprint arXiv:1810.04805},
year={2018}
}