KAN: Kolmogorov–Arnold Networks

Why This Matters

For decades, Multi-Layer Perceptrons (MLPs) have been the undisputed backbone of deep learning. Based on the Universal Approximation Theorem, MLPs guarantee that a neural network can approximate any continuous function by stacking linear operations followed by fixed, non-linear activation functions (like ReLU or SiLU) on nodes.

However, MLPs suffer from several critical shortcomings, particularly in scientific domains:

• The Black Box Problem: Interpreting the weights of an MLP is notoriously difficult, making them unhelpful for scientists trying to extract mathematical formulas or physical laws from data.
• Curse of Dimensionality (COD): As the input dimensionality increases, the number of parameters needed to approximate high-dimensional functions grows exponentially unless specific compositional structures are exploited.
• Catastrophic Forgetting: When trained on sequential tasks, MLPs update global weights, destroying previously learned patterns.

The Big Idea

Instead of using the Universal Approximation Theorem as inspiration, KANs are built on the Kolmogorov-Arnold Representation Theorem. This theorem states that any multivariate continuous function can be represented as a finite composition of continuous functions of a single variable and the binary operation of addition:

The authors generalize this depth-2 representation to arbitrary widths and depths, enabling the creation of deep, trainable architectures. By replacing the fixed activation functions on nodes with learnable univariate functions (B-splines) on the edges, and replacing the linear weight matrices with these non-linear edge functions, KANs completely reimagine how networks learn features.

How It Works

1. Node vs. Edge Computation

In an MLP, information flows along linear weights ($w_{ij}$) and is transformed at the target node by a fixed activation function ($\sigma$). In a KAN, the node simply sums up incoming signals, while the transformation happens along the connection edge itself via a learnable univariate function $\phi(x)$.

2. B-Splines & Residual Activations

To make the edge activations $\phi(x)$ highly flexible and locally adjustable, KANs parameterize them as a combination of a global basis function and a local B-spline:

where $b(x) = \text{silu}(x) = \frac{x}{1 + e^{-x}}$ acts as a residual connection to preserve standard linear behavior, and the spline is a linear combination of B-spline bases:

3. Grid Extension

A key advantage of splines is the ability to perform Grid Extension. Instead of training a massive network from scratch, you can train a KAN with a coarse spline grid (few intervals) and then "fine-grain" the grid to achieve higher precision without retraining from scratch.

4. Sparsification & Symbolification

KANs achieve interpretability through a unique process:

1. Sparsification: Training with $L_1$ and entropy regularization on the activation functions to prune away redundant nodes and edges.
2. Symbolic Snapping: Staring at the learned univariate curves, recognizing their mathematical shape (e.g., $\sin$, $\ln$, $x^2$), and locking them into exact symbolic formulas.

Empirical Results

The paper presents stunning results on small-scale AI + Science tasks, proving KANs' superiority over MLPs in several core dimensions:

Particularly in physics (Anderson localization) and mathematics (knot theory), KANs functioned as collaborative scientific assistants, allowing researchers to easily identify key variables and discover analytical equations from raw numerical data.

Limitations & Open Questions

While KANs are incredibly elegant and powerful for small-scale scientific tasks, the authors highlight several critical bottlenecks:

• Slow Training Speeds: Currently, KANs train roughly 10x slower than MLPs with the same number of parameters. This is because different spline activation functions on edges cannot easily leverage optimized batch computations.
• Hardware Optimization: Modern GPUs are heavily designed for rapid matrix multiplications (perfect for MLPs). KANs require custom kernels to achieve comparable hardware efficiency.
• Scaling to Large Models: It remains an open question whether KANs can scale to massive architectures like LLMs, or if they will remain specialized tools for AI + Science.

Glossary

Citation

@inproceedings{liu2024kan,
  title={KAN: Kolmogorov--Arnold Networks},
  author={Liu, Ziming and Wang, Yixuan and Vaidya, Sachin and Ruehle, Fabian and Halverson, James and Solja{\v{c}}i{\'{c}}, Marin and Hou, Thomas Y and Tegmark, Max},
  booktitle={International Conference on Learning Representations (ICLR)},
  year={2025}
}