Cortex: Consolidating Context into Parametric Memory
Jintian Zhang1,2,*, Feng He2, Xinnian Liang2, Ziwen Xu1, Haoming Xu1, Haojing Huang2,3,*, Shuangzhi Wu2, Ningyu Zhang1,†
1Zhejiang University • 2ByteDance Seed • 3Tsinghua University
TL;DR
Large language models excel when given informative context (APIs, persona rules, or facts), but forget everything the second that context window is flushed. Inspired by biological systems consolidation, Cortex turns transient prompt contexts into permanent parametric knowledge by rehearsing anticipated queries and distilling both output distributions and post-FFN residual states into mergeable neural plasticity channels—achieving an 88.9% consolidation rate and 99.8% retention even after 50 continuous edits.
1 Why This Matters: The Transient Context Trap
Modern AI agents live in an ephemeral present. Provide a language model with detailed documentation for a brand-new internal API, a persona guide, or a correction to an out-of-date fact, and it will immediately execute commands with high precision. But standard inference is strictly immutable: the model parameters $\theta$ remain untouched. Once that session finishes or the context window shifts, the intelligence vanishes.
To date, researchers have addressed this through two main paradigms, both carrying fatal drawbacks:
Relies on prompt retrieval at inference time. If a future task relies on indirectly relevant experience or requires multi-step composition, retrieval fails to surface the document, and context bloat drives latency through the roof.
Algorithms like ROME, MEMIT, or AlphaEdit are tailored narrowly to single factual triples ($s, r, o$). In continual multi-step editing, models quickly degenerate into infinite repetition loops or catastrophic forgetting.
The human brain solved this dilemma eons ago through systems consolidation. In the Complementary Learning Systems (CLS) framework, the hippocampus rapidly absorbs episodic experiences, while the neocortex slowly integrates that knowledge into structural networks via reactivation and replay—allowing permanent retention without disrupting older faculties. Cortex translates this biological architecture directly into the heart of modern Transformer networks.
2 The Big Idea: Memory Consolidation (MC)
Rather than treating factual editing, API tool learning, unlearning, and persona steering as disconnected tasks with distinct optimization tricks, Memory Consolidation formalizes them all under a single objective: internalizing the performance gains $\Delta_{\text{ctx}}$ brought by any temporary context $C$ into persistent Feed-Forward parameters, such that the updated model $M'$ operates seamlessly without $C$ while preserving non-target pre-training capabilities.
Mathematically, for target queries $Q^+_C$ and retain queries $Q^-_C$, the authors measure two complementary coordinates:
Where $\text{CR} = 1.0$ indicates that the context-free student has recovered 100% of the benefit of having the prompt in context, and $\text{RR} = 1.0$ denotes zero degradation of unrelated skills (e.g., MMLU, GSM8K, SimpleQA).
3 How Cortex Works: The Three Pillars
1. What to Learn: Prospective Rehearsal
In biological memory, remembering is prospective: the brain doesn't just store transcripts of the past; it imagines how that experience will be applied to future scenarios.
Cortex achieves this by prompting the frozen pre-update model $M_{t-1}$ to hallucinate future queries $q_i$ that would benefit from context $C$, and pairing them with context-conditioned responses $y_i$:
Crucially, using the model's own self-generated answers rather than external teacher answers prevents catastrophic distribution shifts. The paper shows in Appendix B.3.7 that using external frontier model answers (e.g. Gemini 3 Pro) causes synaptic efficacy to fall from 89.5% to 21.4% and MMLU from 75.8% to 15.9% (Appendix B.3.7, GPT-OSS-20B FFN-only Synaptic editing).
2. Where to Write: FFN Plasticity & Shared MoE Branches
Drawing on discoveries that Transformer Feed-Forward Networks (FFNs) act as associative key-value memories, Cortex confines all parametric writes strictly to FFN sublayers. For a SwiGLU FFN: $$F(x) = W_D[(W_U x) \odot \text{SiLU}(W_G x)]$$ Cortex introduces two biological analogies:
Strengthens or dampens existing synaptic contacts via low-rank updates to existing projections:
Best for: Factual memory edits and local entity updates. Merges directly into base weights.
Sprouts new dendritic spines by adding $k$ nonlinear feature channels while freezing original weight blocks:
Best for: Complex procedural reasoning and new API syntax. Folds into wider FFNs.
The MoE Dilemma: Why Independent Experts Starve
In Mixture-of-Experts (MoE) architectures, adding adapters to individual experts fails because router selections are sparse—in ALFWorld, the authors found 32 out of 768 expert branches received <1% of mean token exposure! Cortex introduces a Shared Structural Branch $E_s(x)$ evaluated for every token: $$F'_{\text{MoE}}(x) = \sum_{e} g_e(x) E_e(x) + E_s(x)$$ Because routing weights satisfy $\sum_e g_e(x) = 1$, at inference time this shared channel can be identically folded into each individual expert: $\sum_e g_e(x)[E_e(x) + E_s(x)]$.
3. How to Update: Joint Distributional KL + Hidden-State MSE
How do we compel a context-free student model to mimic the computation of an explicit-context teacher? Cortex minimizes a dual-loss objective over minibatch positions $J_B$:
- Truncated Logit KL: Uses top-$K_v = 1024$ teacher logits at temperature $T=2$. Prevents memorization of noise while transferring predictive entropy.
- Post-FFN Residual MSE: When context is removed, attention states change drastically. Appendix A mathematically demonstrates that an additive FFN update can exactly compensate for this discrepancy if intermediate post-FFN residual states $h_{\ell,i}$ are aligned layer-by-layer!
- Zero-Context Historical Query Replay: During sequential updates, Cortex avoids storing past contexts. It stores only raw query strings $q$, and regenerates new teacher targets on the fly using $M_{t-1}$.
FFN Plasticity Architecture Explorer
Adjust the slider below to inspect parameter counts, memory footprints, and architectural modifications across model layers ($d_h = 5120$, $d_{ff} = 13824$ typical for a 14B model).
Continual 50-Fact Sequential Editing Simulator (Illustrative)
Illustrative simulation with made-up curves, not paper data; the only paper data point is the final step-50 checkpoint. Slide the update step from Fact 1 to Fact 50. Watch how traditional editing methods suffer catastrophic forgetting, whereas Cortex maintains early facts through regenerator replay!
4 Empirical Results & The CR-RR Pareto Frontier
The authors systematically evaluated Cortex across models from 4B to 120B parameters—including instruction-tuned, thinking (reasoning), dense, and MoE backbones (Qwen2.5-14B, Qwen3-4B-Thinking, Gemma3-12B, GPT-OSS-20B, and GPT-OSS-120B).
Across 15 task × model settings (the panels of Fig. 5) spanning the 5 task families, Cortex occupies the empirical Pareto frontier in most evaluated settings, outperforming specialized baselines:
Benchmark Pareto Frontier Explorer
| Method | Efficacy (%) | Portability (%) | Locality (%) | Consolidation Rate (CR) | Retention Rate (RR) | Params Added |
|---|---|---|---|---|---|---|
| Vanilla Reference | 2.9% | 19.1% | 40.3% | 0.0% | 100.0% | 0 M |
| FT-M | 22.8% | 17.8% | 32.7% | 12.6% | 95.0% | 70.8 M |
| ROME (Meng et al.) | 26.2% | 39.0% | 32.2% | 28.5% | 94.4% | 70.8 M |
| MEMIT (Meng et al.) | 22.9% | 36.5% | 35.1% | 23.2% | 96.4% | 353.9 M |
| AlphaEdit (Fang et al.) | 21.6% | 35.6% | 34.8% | 22.7% | 96.0% | 353.9 M |
| WISE (Wang et al.) | 34.3% | 30.9% | 39.7% | 28.0% | 98.9% | 70.8 M |
| Cortex (Synaptic r=1) | 95.6% | 30.7% | 33.8% | 80.6% | 97.2% | 2.73 M/edit |
| Cortex (Structural k=1) | 95.3% | 49.4% | 38.3% | 88.9% | 99.8% | 0.74 M/edit |
Key Scientific Insights from the Ablations:
- Attention Updates Harm Retention (RQ2): When extending updates from FFNs to Self-Attention projections (LoRA-All), parameters jumped from 10.9M to 17.2M, but CR dropped by 3.8 points and RR dropped by 4.0 points. FFN-only updates preserve non-target skills far better.
- Structural Expansion Excels in Compositional Portability: Structural achieves 49.4% portability vs 30.7% for Synaptic. Why? Adding orthogonal nonlinear channels allows the model to compute composite multi-step functions without destroying existing key-value pathways.
- Context Prompting is Not an Upper Bound: In ALFWorld agent tasks, consolidated models reached 128–133% CR, outperforming in-context prompting itself! Distilling experience cleans up messy episodic noise, allowing the agent to execute actions faster (15.6 vs 18.8 steps).
5 Limitations & Open Questions
Local Knowledge Suppression Bleed
In knowledge unlearning (RWKU), suppressing an entity caused a mean 9–20 point drop on neighboring knowledge probes, even though general BBH accuracy remained pristine. Precise surgical forgetting without neighborhood collateral damage remains unsolved.
Offline Trajectory vs. On-Policy Shift
Cortex optimizes using teacher forcing on fixed, offline trajectories generated by $M_{t-1}$. In highly dynamic environments with long rollouts (e.g., $\tau^2$-Bench telecom tasks), student policy drift occasionally causes consolidation failure (CR fell to -92.3% on Synaptic GPT-OSS-20B).
Accumulation & Future Reorganization
While Structural plasticity adds only 0.74M parameters per edit ($k=1$), after 5,000 continuous edits, the added channels will swell the intermediate width by $5000 \times k$. Biological brains alternate between expansion and sleep-phase pruning; future work must investigate structural compression and channel merging.
6 Glossary
Memory Consolidation (MC)
The objective of converting transient in-context behavioral gains ($\Delta_{\text{ctx}}$) into permanent, context-free model parameters while preserving capabilities outside the scope of the change.
Consolidation Rate (CR)
$\frac{\text{Perf}(M', Q^+_C) - \text{Perf}(M, Q^+_C)}{\text{Perf}(M, Q^+_C, C) - \text{Perf}(M, Q^+_C)}$. Represents the percentage of the context's prompt-injected performance recovered by the model when evaluated completely without context.
Retention Rate (RR)
$\frac{\text{Perf}(M', Q^-_C)}{\text{Perf}(M, Q^-_C)}$. Measures whether general capabilities (reasoning on GSM8K, factual knowledge on SimpleQA/MMLU, coding on LiveCodeBench) were degraded by the weight updates.
Structural Plasticity
Adding new parallel nonlinear neurons/channels ($U_s, G_s, D_s$) directly to the SwiGLU FFN layer while leaving base weight matrices completely frozen. Easily merged into an expanded layer width $d_{ff} + k$.
7 BibTeX Citation
@article{zhang2026cortex,
title = {Cortex: Consolidating Context into Parametric Memory},
author = {Jintian Zhang and Feng He and Xinnian Liang and Ziwen Xu and Haoming Xu and Haojing Huang and Shuangzhi Wu and Ningyu Zhang},
journal = {arXiv preprint arXiv:2610.20399},
year = {2026},
url = {https://arxiv.org/abs/2610.20399}
}