Why This Matters
Large Language Models (LLMs) have traditionally scaled performance by scaling parameter count. However, running a dense model like LLaMA 2 70B requires massive computational resources and incurs high latency during generation. Every single token must pass through every single parameter.
Mixtral 8x7B breaks this paradigm by using a Sparse Mixture of Experts (SMoE). It offers the capacity of a 47-billion-parameter model but only utilizes 13 billion active parameters per token. This enables exceptionally fast inference and high throughput while matching or exceeding the capabilities of much larger, closed-source models.
The Big Idea
Instead of a single Feed-Forward Network (FFN) block in each transformer layer, Mixtral implements 8 parallel "experts". For every incoming token, a lightweight router network selects the top 2 experts to process the token. The outputs of these two selected experts are then combined using a weighted sum.
Because the routing decisions are made dynamically per token, different tokens in a single sentence can route through completely different expert pathways. This allows the model to partition knowledge and computation efficiently.
How It Works
The mathematical framework of Mixtral is elegant. Let $n$ be the number of experts (8 for Mixtral), and let $E_i(x)$ be the output of the $i$-th expert network for a given input token representation $x$.
The output $y$ of the Mixture-of-Experts layer is computed as:
Where $G(x)_i$ is the routing weight assigned to expert $i$. If $G(x)_i = 0$, we bypass computing $E_i(x)$ entirely, saving massive amounts of computational power.
To achieve this sparsity, the router selects the top $K$ logits (where $K=2$ in Mixtral) and applies a softmax over them:
Where $W_g$ represents the trainable gating weights, and the $\text{TopK}$ function sets all values outside the top $K$ to $-\infty$ before computing the softmax.
Interactive Routing Simulator
Select a token type or manually adjust the router's raw affinity score (logits) for each of the 8 experts. See how the Top-2 selection dynamically routes the token and computes the final Softmax weights $G(x)_i$.
MoE Parameter & Cost Calculator
How does changing the total number of experts ($N$) and the active experts per token ($K$) impact model size and compute requirements? Use the sliders below to explore the trade-offs.
Results & Benchmarks
Mixtral 8x7B was evaluated against LLaMA 2 70B, LLaMA 2 13B, and GPT-3.5 on popular benchmarks. Despite having only 13B active parameters, Mixtral comfortably beats or matches LLaMA 2 70B and GPT-3.5 across the board.
*Data sourced directly from Tables 2 and 3 of the Mixtral of Experts paper.
Limitations & Open Questions
Despite its strong performance, Mixture of Experts models introduce unique challenges:
- VRAM Footprint: While Mixtral only uses 13B active parameters per token, the entire 47B parameter model must still be loaded into GPU memory (VRAM) for high-speed inference. This makes local hosting on consumer hardware more difficult than a dense 13B model.
- Expert Specialization Limits: The routing analysis in Section 5 shows that experts do not specialize cleanly into semantic domains (e.g., one expert for biology, one for math). Instead, routing decisions appear heavily influenced by syntax and local context (consecutive tokens often route to the same expert).
- Load Balancing: Distributing MoE models across multiple GPUs (Expert Parallelism) requires careful load balancing to ensure specific GPUs aren't bottlenecked by popular experts.
Glossary
Sparse Mixture of Experts (SMoE)
A neural network architecture where only a subset of the model's parameters (the "experts") are activated for any given input, allowing for massive model capacity without a proportional increase in computational cost.
Router Network / Gating Network
A small feed-forward network that processes the input token representation and produces a probability distribution over the available experts, determining which experts should process the token.
Active Parameters
The subset of parameters that are actually computed during a single forward pass for a given token. For Mixtral, this is approximately 13 billion parameters out of 47 billion total.
Cite this Paper
@article{jiang2024mixtral,
title={Mixtral of Experts},
author={Jiang, Albert Q and Sablayrolles, Alexandre and Roux, Antoine and Mensch, Arthur and Savary, Blanche and Bamford, Chris and Chaplot, Devendra Singh and de las Casas, Diego and Hanna, Emma Bou and Bressand, Florian and others},
journal={arXiv preprint arXiv:2401.04088},
year={2024}
}