Contrastive Learning — Concept Reference

In One Sentence

Contrastive learning is a self-supervised representation learning paradigm that trains models to map semantically similar ("positive") pairs close together in an embedding space, while pushing dissimilar ("negative") pairs far apart.

Intuition

Imagine trying to teach someone what a "dog" looks like without ever using the word "dog" or showing them labeled categories. Instead, you show them two different photos of the same golden retriever (perhaps one cropped, one black-and-white) and say: "These two are the same thing." Then, you show them a photo of a golden retriever alongside a photo of a bicycle, a teapot, or a cat, and say: "These two are different things."

By repeating this process millions of times, the learner naturally discovers the underlying structural features that make a dog a dog, ignoring irrelevant details like background color, lighting, or orientation.

In machine learning, we operationalize this using data augmentations. For any given image, we apply random transformations (like cropping, color jittering, or blurring) to create two "views" of the same source. Because they originate from the same image, they form a positive pair. Views generated from different images in the dataset form negative pairs. The model's objective is to maximize the similarity of positive pairs while minimizing the similarity of negative pairs.

Formal Definition

The standard objective function used in modern contrastive learning is the InfoNCE Loss (often formulated as the NT-Xent loss in frameworks like SimCLR).

Let $x_i$ and $x_j$ be a positive pair of augmented samples. Let $z_i = g(f(x_i))$ and $z_j = g(f(x_j))$ be their corresponding low-dimensional representations, where $f(\cdot)$ is a backbone encoder (e.g., ResNet) and $g(\cdot)$ is a non-linear projection head.

For a batch containing a positive pair $(i, j)$ and a set of negative samples, the loss for the positive pair is defined as:

$$ \mathcal{L}_{i,j} = -\log \frac{\exp(\text{sim}(z_i, z_j) / \tau)}{\sum_{k=1}^{2N} \mathbb{I}_{[k \neq i]} \exp(\text{sim}(z_i, z_k) / \tau)} $$

Where:

$\text{sim}(u, v) = \frac{u^T v}{\|u\| \|v\|}$ is the cosine similarity between two normalized vectors.
$\tau$ is a temperature parameter that scales the input similarities, controlling the penalty scaling for hard negative samples.
$N$ is the minibatch size (the denominator sums over all $2N$ augmented views in the batch, excluding the anchor itself).

Interactive Walkthrough

Drag the Positive (P) and Negative (N) representations in the embedding space relative to the Anchor (A). Observe how the cosine similarities and the resulting InfoNCE loss change dynamically.

Drag P or N to reposition

Parameters

Temperature ($\tau$) 0.5

Metrics

Similarity $\text{sim}(A, P)$ 0.80

Similarity $\text{sim}(A, N)$ -0.30

InfoNCE Loss 0.126

When to use & when not to

When to use

Abundant Unlabeled Data: Highly effective for pre-training encoders on massive datasets where human labels are expensive or unavailable.
Multi-modal Alignment: Ideal for cross-modal tasks (e.g., matching text descriptions to images, as demonstrated by CLIP).
Downstream Transferability: Learned representations capture rich semantic structures, often outperforming supervised counterparts on downstream tasks.

When not to use

Extreme Compute Constraints: Standard contrastive learning (like SimCLR) relies on massive batch sizes (e.g., 4096) to provide enough negative samples, requiring significant GPU memory.
Ambiguous Augmentations: If semantic meaning is destroyed by standard transformations (e.g., medical imaging where color or orientation determines a pathology), standard contrastive pipelines can fail.

Key Milestones & Evolution

Momentum Contrast (MoCo)

Late 2019

Introduced by He et al. (arXiv:1911.05722), MoCo framed contrastive learning as a dynamic dictionary look-up. By utilizing a queue to store negative keys and a momentum-updated target encoder, it decoupled dictionary size from minibatch size, enabling efficient contrastive learning.

SimCLR Framework

Early 2020

Chen et al. (arXiv:2002.05709) simplified self-supervised visual representation learning. They proved that a simple end-to-end framework with strong data augmentations, a non-linear projection head, and large batch sizes could match supervised ResNet-50 performance.

Bootstrap Your Own Latent (BYOL)

Mid 2020

Grill et al. (arXiv:2006.07733) challenged the necessity of negative pairs. By training an online network to predict the target network's representation of an augmented view (with the target updated via a slow-moving average), BYOL achieved state-of-the-art results without relying on negative samples.

CLIP (Contrastive Language-Image Pre-training)

Early 2021

Radford et al. (arXiv:2103.00020) scaled contrastive learning to multi-modal domains. By training on 400 million image-text pairs to predict which caption matches which image, they demonstrated powerful zero-shot generalization to diverse downstream tasks.

Pseudocode

simclr_loss.py PyTorch Implementation

import torch
import torch.nn as nn
import torch.nn.functional as F

class SimCLRLoss(nn.Module):
    def __init__(self, batch_size, temperature=0.5):
        super().__init__()
        self.batch_size = batch_size
        self.temperature = temperature
        # Mask to exclude self-similarity from denominator
        self.mask = ~torch.eye(2 * batch_size, dtype=torch.bool)

    def forward(self, z_i, z_j):
        """
        z_i, z_j: [batch_size, dim] representations of two augmented views
        """
        # Concatenate views to get a [2*B, dim] tensor
        representations = torch.cat([z_i, z_j], dim=0)
        
        # Compute pairwise cosine similarity matrix [2*B, 2*B]
        similarity_matrix = F.cosine_similarity(
            representations.unsqueeze(1), 
            representations.unsqueeze(0), 
            dim=2
        )
        
        # Extract positive similarity scores
        sim_ij = torch.diag(similarity_matrix, self.batch_size)
        sim_ji = torch.diag(similarity_matrix, -self.batch_size)
        positives = torch.cat([sim_ij, sim_ji], dim=0) # [2*B]
        
        # Filter out self-similarity from denominator
        negatives = similarity_matrix[self.mask].view(2 * self.batch_size, -1)
        
        # Combine positives and negatives: positive is at index 0
        logits = torch.cat([positives.unsqueeze(1), negatives], dim=1)
        logits /= self.temperature
        
        # Cross entropy loss against target index 0
        labels = torch.zeros(2 * self.batch_size, dtype=torch.long, device=z_i.device)
        return F.cross_entropy(logits, labels)

Common Variants

End-to-End (SimCLR)

Relies on massive batch sizes to guarantee a rich set of negative samples. Uses aggressive data augmentation and a non-linear projection head to map representations to a hypersphere.

Memory-Queue (MoCo)

Maintains a continuous queue of negative keys. Updates a key encoder using a momentum-based moving average of the query encoder, allowing large dictionary sizes without huge batch sizes.

Negative-Free (BYOL)

Eliminates negative pairs completely. Uses an online network to predict the target network's representation of an augmented view, avoiding representation collapse via architectural asymmetry.

Multi-Modal (CLIP)

Applies contrastive learning to distinct modalities (e.g., text and images). Embeddings of corresponding image-text pairs are pulled together, while non-matching pairs are pushed apart.

Related Core Architectures

Deep Residual Learning (ResNet)

ResNet-50 serves as the standard, foundational backbone encoder for modern self-supervised contrastive learning methods including SimCLR, MoCo, and BYOL.

Vision Transformers (ViT)

Transformers adapted for vision tasks are increasingly used as the primary backbone in state-of-the-art scaled contrastive models like CLIP.