Security Vulnerability ICML 2024 Model Stealing

Stealing Part of a Production Language Model

By Nicholas Carlini, Daniel Paleka, Krishnamurthy Dvijotham, Thomas Steinke, Jonathan Hayase, and colleagues from Google DeepMind, ETH Zurich, UW, OpenAI, & McGill.

TL;DR The first high-fidelity model-stealing attack against black-box production LLMs (like GPT-4 and PaLM-2). By exploiting standard API features like logit bias or top-K logprobs, the attack extracts the model's exact hidden dimension and reconstructs its final embedding projection matrix for as little as $20 USD.

Vulnerable Targets

GPT-4, PaLM-2, Ada

Extraction Cost

< $20 (Ada) / $200 (GPT-3.5)

Reconstruction Error

10⁻⁴ MSE (Up to symmetry)

Why This Matters

Large language model providers keep their model architectures, parameter counts, and weights strictly confidential. This secrecy is justified by competitive advantages and safety guardrails. Even when weights are sealed behind cloud APIs, researchers assumed the internal parameters were secure.

This paper shatters that assumption. It presents a practical, low-cost attack that extracts the exact weights of the final layer of a production transformer.

Knowing the final projection layer (unembedding matrix) reveals the model's exact hidden dimension width ($h$), which is highly correlated with total parameter count. It also makes the model less of a "black box," potentially opening the door to more sophisticated downstream attacks.

The Big Idea

The core insight relies on a simple mathematical truth: the final layer of a language model is low-rank.

A transformer's final linear layer projects a relatively small hidden dimension vector $h$ (typically 1,024 to 8,192) up to a massive vocabulary space $l$ (often 32,000 to over 100,000 tokens).

Because $h \ll l$, any set of output logit vectors generated by the model must reside in an $h$-dimensional subspace within the larger $l$-dimensional vocabulary space. By querying the model with various inputs and collecting the resulting outputs, an attacker can use Singular Value Decomposition (SVD) to isolate this subspace and reconstruct the projection matrix $W$ up to a linear transformation.

How It Works

The attack progresses from a theoretical "all-logits" setup to bypassing real-world API restrictions like logit bias constraints and missing logprobs.

Step 1: Discovering the Hidden Dimension ($h$)

Suppose we query the model with $n$ random prompt prefixes. The API returns a logit matrix $Q \in \mathbb{R}^{l \times n}$. Since $Q = W \cdot H$, where $H \in \mathbb{R}^{h \times n}$ contains the model's final hidden states:

$$\text{rank}(Q) \le \min(l, n, h) = h \quad (\text{if } n > h)$$

By plotting the singular values of $Q$, we expect a sharp cliff exactly at index $h$. Any singular values beyond $h$ are purely numerical noise from floating-point imprecision.

1

Interactive SVD Subspace Identifier

Simulates real-world SVD extraction

True Hidden Dimension ($h$)

256

Number of API Queries ($n$)

300

Floating-Point Precision

💡 Observe: If $n < h$, the graph shows no sharp drop-off because the matrix is full-rank. Once $n > h$, a dramatic cliff appears at index $h$.

⚠️ Bfloat16 / Noise: Higher noise levels blur the cliff, making exact identification harder.

Detected h: 256

Singular Value Index (1 to 600) Magnitude (Log Scale)

Step 2: Exploiting the Logit Bias API

In production environments, APIs do not return raw logits. Instead, they return the top-K logprobs. To extract raw logits, the authors exploited the logit bias parameter, which allows users to artificially shift logits before the softmax operation:

$$y_i^B = z_i + B - \log \sum_{j \neq i} \exp(z_j) + \exp(z_i + B)$$

By adding a massive bias $B$ to a target token $i$, we force it into the top-K outputs. Comparing it against a stable "reference" token $R$ allows us to extract the precise difference between any two logits:

$$y_R^B - y_i^B - B = z_R - z_i$$

2

Logit Bias Extraction Simulator

Reconstructs hidden logits in real-time

Set "true" hidden logits for 4 vocabulary tokens. Use the bias slider to see how applying bias exposes hidden values through relative measurements.

Token A Logit ($z_A$)

Token B Logit ($z_B$)

Token C Logit ($z_C$)

Token D Logit ($z_D$) [Reference] Locked as base reference (0.0)

Applied Logit Bias ($B$)

8

API Output (Top-2 Logprobs)

Reconstructed Logits

Token A

-

Token B

-

Token C

-

Step 3: Distinguishing LayerNorm vs RMSNorm

Different normalization layers leave unique mathematical footprints. LayerNorm centers its outputs, projecting them onto a $(h-1)$-dimensional subspace. RMSNorm does not center, preserving the full $h$ dimensions.

By applying a simple subtraction trick on the extracted logits: $$\tilde{Q} = Q_i - \frac{1}{n}\sum Q_j$$ We can cancel out the LayerNorm bias. If the model uses LayerNorm, the rank of the subtracted matrix drops by exactly 1. If it uses RMSNorm, the rank remains completely unchanged.

3

Normalization Layer Signature Detector

Detects architectural choices passively

Toggle between LayerNorm and RMSNorm. Observe how subtracting the mean logits reveals the centering projection of LayerNorm.

Theoretical Dimension ($h$): 512

Effective Rank (Standard): 512

Effective Rank (Subtracted): 511

Singular Value Cliff Zoom-In

Index 500 Drop at 511 (LayerNorm) vs 512 (RMSNorm) Index 520

Empirical Results

The authors evaluated their attack on several production models. They successfully extracted exact hidden dimensions and reconstructed the final projection layers with extremely high fidelity.

Model Name	True Dim ($h$)	Stolen Dim	W RMS Error	Query Cost	Estimated Cost (USD)
OpenAI ada	1,024	1024 ✓	5 · 10⁻⁴	< 2 · 10⁶	$1
OpenAI babbage	2,048	2048 ✓	7 · 10⁻⁴	< 4 · 10⁶	$2
babbage-002	1,536	1536 ✓	N/A	< 4 · 10⁶	$12
gpt-3.5-turbo-instruct	Confidential	Confirmed ✓	N/A	< 4 · 10⁷	$200

Limitations & Defenses

While highly effective, the attack has clear boundary conditions and straightforward mitigations.

Limitations

Single-Layer Boundary: The attack only recovers the final projection layer. Reconstructing deeper layers is blocked by non-linear activation functions.
Symmetry Invariance: The weight matrix $W$ is recovered up to a linear transformation ($W \cdot G$), not bit-for-bit exactly.

Defenses

Logit Bias XOR Logprobs: Prohibiting APIs from supplying logit biases while returning logprobs makes the attack 10× more expensive.
Noise Injection: Adding small Gaussian noise to output logits degrades SVD precision, shielding the subspace.
Removing Logit Bias: Completely deprecating logit bias prevents the attack entirely, though it breaks some downstream developer features.

Glossary

Unembedding Matrix ($W$)

The final linear projection layer in a transformer model. It maps the low-dimensional hidden state vector ($h$) back to the high-dimensional vocabulary space ($l$) to generate token probabilities.

Singular Value Decomposition (SVD)

A fundamental matrix factorization method that decomposes a matrix into singular vectors and singular values. In this paper, SVD is used to identify the effective rank ($h$) of the output logit space.

Logit Bias

An API parameter that allows developers to modify the likelihood of specified tokens appearing in the completion by adding a user-defined constant to their raw logits before the softmax step.

Citation

@inproceedings{carlini2024stealing,
  title={Stealing Part of a Production Language Model},
  author={Carlini, Nicholas and Paleka, Daniel and Dvijotham, Krishnamurthy and Steinke, Thomas and Hayase, Jonathan and Cooper, A. Feder and Lee, Katherine and Jagielski, Matthew and Nasr, Milad and Conmy, Arthur and Yona, Itay and Wallace, Eric and Rolnick, David and Tram{\`e}r, Florian},
  booktitle={International Conference on Machine Learning (ICML)},
  year={2024}
}