Sequence to Sequence Learning with Neural Networks

Why This Matters

Before 2014, Deep Neural Networks (DNNs) were making massive waves in computer vision and speech recognition. Yet, they harbored a massive Achilles' heel: they could only process inputs and outputs of fixed dimensionality.

Real-world problems like Machine Translation, question answering, and speech dialog are inherently sequential, with input and output lengths that are highly variable and completely unknown a-priori. Mapping a 10-word English sentence to a 12-word French sentence with arbitrary, non-monotonic word alignments was incredibly difficult for standard neural architectures to solve end-to-end.

The Big Idea

The paper proposes a remarkably elegant solution: use two distinct LSTMs.

"Our method uses a multilayered Long Short-Term Memory (LSTM) to map the input sequence to a vector of a fixed dimensionality, and then another deep LSTM to decode the target sequence from the vector."

This forms the classic Encoder-Decoder architecture. The Encoder reads the input sequence token-by-token and compresses it into a single, high-dimensional vector representation (often called the context vector). The Decoder then acts as a conditional language model, generating the output sequence token-by-token, starting directly from that context vector.

How It Works

1. Mathematical Formulation

The goal of the network is to estimate the conditional probability $p(y_1, \dots, y_{T'} | x_1, \dots, x_T)$, where the input sequence length $T$ and output sequence length $T'$ can differ.

The Encoder first computes a fixed-dimensional representation $v$ of the input sequence $(x_1, \dots, x_T)$ using its final hidden state. The Decoder then computes the probability of the target sequence:

$$p(y_1, \dots, y_{T'} | x_1, \dots, x_T) = \prod_{t=1}^{T'} p(y_t | v, y_1, \dots, y_{t-1})$$

Each $p(y_t | v, y_1, \dots, y_{t-1})$ distribution is represented by a softmax over all the words in the vocabulary.

1 Interactive Widget: The Power of Source Reversal

The authors found that reversing the source sentence (e.g., "A B C" to "C B A") dramatically improved translation accuracy. Why? It introduces many short-term dependencies, making it much easier for backpropagation to "establish communication" between the source and target. Toggle below to visualize the "minimal time lag" difference.

In Normal Order, the word "A" is 4 steps away from its translation "X".

2 Interactive Beam Search Simulator

The model uses a left-to-right beam search decoder to find the most likely translation. Adjust the beam size $B$ and step through the generation process to see how paths are pruned.

Beam Size (B):

Step 1 (Start)

Step 2

Step 3

Step 4 (End)

3 Interactive Semantic Space Explorer

The LSTM learns variable-length sentence representations that are highly sensitive to word order and invariant to active/passive voice. Hover over the points to inspect the learned semantic structure (recreated from Figure 2 of the paper).

Selected Sentence

Hover over any node in the chart to inspect.

Semantic Cluster Match

None

Results & Performance

On the WMT'14 English-to-French translation task, the deep LSTM models achieved unprecedented success for pure neural network architectures.

WMT'14 English-to-French BLEU Scores

Phrase-based SMT Baseline (Durrani et al.) 33.30 BLEU

Single Forward LSTM (Beam Size 12) 26.17 BLEU

Single Reversed LSTM (Beam Size 12) (+4.42 improvement!) 30.59 BLEU

Ensemble of 5 Reversed LSTMs (Beam Size 12) 34.81 BLEU

SMT Baseline + Ensemble Rescoring (Best Result) 36.50 BLEU

Note: The LSTM's BLEU score of 34.81 was penalized on out-of-vocabulary words because of its fixed 80k target word vocabulary. Despite this, it comfortably outperformed the phrase-based SMT baseline.

Limitations & Open Questions

The "Information Bottleneck": Forcing an entire sentence of arbitrary length into a single, fixed-size vector representation ($v$) is a huge bottleneck. While the paper claims the model did not suffer on long sentences, later research proved this bottleneck severely restricts scaling (which eventually led to the development of Attention Mechanisms).
Fixed Vocabulary Constraints: Out-of-vocabulary (OOV) words were mapped to a generic UNK token. This penalized performance on rare words or names.
Why Reversal Actually Works: The authors admit they don't have a complete mathematical explanation for why reversing the input helps so much, but hypothesize it centers entirely around reducing the minimal time lag for gradient propagation.

Glossary

BLEU (Bilingual Evaluation Understudy)

An algorithm for evaluating the quality of text which has been machine-translated from one natural language to another. Scores are calculated for individual translated segments by comparing them with a set of good quality reference translations.

Beam Search

A heuristic search algorithm that explores a graph by expanding the most promising nodes in a limited set. In machine translation, it keeps the top $B$ most likely translation prefixes at each step to avoid the exponential complexity of searching the entire vocabulary tree.

Perplexity

A measurement of how well a probability distribution or probability model predicts a sample. A low perplexity indicates the probability distribution is good at predicting the sample.

Citation

@inproceedings{sutskever2014sequence,
  title={Sequence to sequence learning with neural networks},
  author={Sutskever, Ilya and Vinyals, Oriol and Le, Quoc V},
  booktitle={Advances in neural information processing systems},
  pages={3104--3112},
  year={2014}
}