Model Compression Landmark

The Era of 1-bit LLMs:
All Large Language Models are in 1.58 Bits

Authors: Shuming Ma*, Hongyu Wang*, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei (Microsoft Research, UCAS)
Published: Feb 2024 (arXiv)

TL;DR

"We introduce BitNet b1.58, a 1-bit LLM variant where every single weight is ternary $\{-1, 0, 1\}$. It matches full-precision (FP16) Transformers in performance starting at 3B parameters, while dramatically slashing latency, memory, and energy consumption."

71.4x Energy Savings 3.55x Memory Reduction 4.1x Decoding Speedup Ternary Weighting

Why This Matters

Modern Large Language Models (LLMs) are computational behemoths. A standard 70-billion parameter model requires hundreds of gigabytes of high-bandwidth memory (HBM) just to load the weights. During inference, every single token generation requires loading all of these weights from memory to the processor, creating a severe memory-bandwidth bottleneck.

Furthermore, standard models perform billions of high-precision floating-point multiplications (FP16 or BF16) for matrix operations. These operations are incredibly energy-intensive and require expensive, specialized hardware like GPUs or TPUs.

BitNet b1.58 breaks this paradigm. By constraining weights to just three values—$-1$, $0$, and $1$—it converts expensive matrix multiplications into simple, low-energy integer additions. This opens up the possibility of running massive LLMs on CPU-based edge devices, mobile phones, and highly optimized, low-cost custom hardware.

The Big Idea

While previous 1-bit models like the original BitNet used binary weights $\{-1, 1\}$, BitNet b1.58 introduces a third value: $0$. This requires $\log_2(3) \approx 1.58$ bits of storage per parameter.

This seemingly small addition of $0$ acts as an explicit feature filter, allowing the network to completely ignore unimportant features. This single modification enables the model to match the perplexity and downstream performance of standard FP16 models, completely bypassing the performance degradation typical of ultra-low-bit quantization.

How It Works

The core of BitNet b1.58 is the replacement of standard linear layers with BitLinear layers. This layer quantizes weights to ternary values and quantizes activations to 8-bit precision.

1. The Ternary Quantization Function

To constrain weights to $\{-1, 0, 1\}$, the model uses an absmean quantization function. It scales the weight matrix by its average absolute value ($\gamma$) and rounds each weight to the nearest integer:

$$\widetilde{W} = \text{RoundClip}\left(\frac{W}{\gamma + \epsilon}, -1, 1\right)$$ $$\text{RoundClip}(x, a, b) = \max(a, \min(b, \text{round}(x)))$$ $$\gamma = \frac{1}{nm} \sum_{ij} |W_{ij}|$$

Interactive Quantization Sandbox

Adjust the continuous weights in the matrix below to see how the absmean quantization maps them to ternary values $\{-1, 0, 1\}$.

Continuous Weights ($W$)

Mean Absolute Scale ($\gamma$): 0.00

Ternary Weights ($\widetilde{W}$)

2. The Computation Paradigm: Multiplication-Free

In a standard model, computing $Y = W \cdot X$ requires floating-point multiply-accumulate (MAC) operations. In BitNet b1.58, because weights are restricted to $\{-1, 0, 1\}$, the computation simplifies to pure additions and subtractions.

FP16 vs. BitNet b1.58 Computation

Click the steps below to see how a dot product is calculated differently. Notice how BitNet eliminates multiplications entirely.

Standard FP16 Requires MAC units

Weights (FP16): [0.29, -0.04, 0.28, -0.18]

Input (FP16): [1.20, 0.85, -0.40, 2.10]

Math:

(0.29 * 1.20) + (-0.04 * 0.85) + (0.28 * -0.40) + (-0.18 * 2.10) = 0.348 - 0.034 - 0.112 - 0.378 = -0.176

BitNet b1.58 Addition-only

Weights (Ternary): [1, 0, 1, -1]

Input (INT8): [12, 8, -4, 21]

Math:

(+12) + (0) + (-4) - (21) = 12 - 4 - 21 = -13

Empirical Results

The paper compares BitNet b1.58 against standard FP16 LLaMA models across various scales. The findings show a clear Pareto improvement: BitNet matches or exceeds full-precision baselines while using far fewer resources.

Scaling & Cost Simulator

Select a model size to simulate real-world inference efficiency gains.

GPU Memory

3.55x less memory

LLaMA FP16: 7.89 GB

BitNet b1.58: 2.22 GB

Latency Speedup

2.71x faster decoding

LLaMA FP16: 5.07 ms

BitNet b1.58: 1.87 ms

Energy Savings

21.7x less energy

Arithmetic Ops: INT8 Additions

vs. Baseline: FP16 MACs

Model Size	Type	Perplexity (PPL) ↓	Zero-Shot Avg Accuracy ↑
3B	LLaMA LLM (FP16)	10.04	49.7%
3B	BitNet b1.58	9.91	50.2%
70B	LLaMA LLM (FP16)	-	333 tokens/s
70B	BitNet b1.58	-	2977 tokens/s (8.9x)

Limitations & Open Questions

Small-Scale Gap: At smaller model sizes (e.g., 700M parameters), there remains a slight perplexity gap between BitNet b1.58 and the FP16 baseline. The scaling benefits only fully crystallize starting at 3B parameters.
Hardware Co-design: Existing off-the-shelf GPUs are highly optimized for floating-point operations. To unlock the full theoretical potential of BitNet b1.58 (especially the 71.4x energy savings), custom hardware and ASICs designed specifically for ternary calculations are required.
Activation Quantization Limits: While weights are aggressively quantized to 1.58 bits, activations are still kept at 8-bit precision. Further research is required to losslessly compress activations to even lower bit depths.

Glossary

Ternary Weights

Weights that can only take on one of three distinct values: -1, 0, or 1. This is in contrast to standard binary weights (-1 and 1) or continuous floating-point weights.

Perplexity (PPL)

A measurement of how well a probability model predicts a sample. In NLP, lower perplexity indicates that the model is more confident and accurate in predicting the next token.

MAC (Multiply-Accumulate)

A common hardware operation that computes the product of two numbers and adds that product to an accumulator. It is the core mathematical operation of deep learning, but is highly energy-intensive compared to simple addition.

Citation

@article{ma2402era,
  title={The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits},
  author={Ma, Shuming and Wang, Hongyu and Ma, Lingxiao and Wang, Lei and Wang, Wenhui and Huang, Shaohan and Dong, Li and Wang, Ruiping and Xue, Jilong and Wei, Furu},
  journal={arXiv preprint arXiv:2402.17764},
  year={2024}
}