Self-Supervised Learning
cs.LG · 70 papers
LoRA: Low-Rank Adaptation of Large Language Models
WikiSQL: 74.0%Edward J. Hu, Yelong Shen +6
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train…
Denoising Diffusion Probabilistic Models (DDPM)
CIFAR10: 9.46Jonathan Ho, Ajay Jain, Pieter Abbeel
We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium…
Spurious Rewards: Rethinking Training Signals in RLVR
MATH-500: 70.8%Rulin Shao, Shuyue Stella Li +12
We show that reinforcement learning with verifiable rewards (RLVR) can elicit strong mathematical reasoning in certain language models even with spurious rewards that have…
s1: Simple Test-Time Scaling
AIME24: 56.7%Niklas Muennighoff, Zitong Yang +8
Test-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance. Recently, OpenAI's o1 model showed this capability…
Generative Adversarial Nets
MNIST: 225 ± 2Ian J. Goodfellow, Jean Pouget-Abadie +6
We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data…
Inside MuZero: Mastering the World Without Knowing the Rules
Atari 57 games (large data): 2041.1%Julian Schrittwieser, Ioannis Antonoglou +10
Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge…
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Wikipedia: 15%Tri Dao, Daniel Y. Fu +3
Transformers are slow and memory-hungry on long sequences, since the time and memory complexity of self-attention are quadratic in sequence length. Approximate attention methods…
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Language Modeling: 5xAlbert Gu, Tri Dao
Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.…
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Reddit TL;DR: 61.0%Rafael Rafailov, Archit Sharma +4
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the…
Proximal Policy Optimization (PPO)
Average Normalized Score: 0.82John Schulman, Filip Wolski +3
We propose a new family of policy gradient methods for reinforcement learning, which alternate between sampling data through interaction with the environment, and optimizing a…
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL
AIME 2024: 79.8%DeepSeek-AI, Daya Guo +198
General reasoning represents a long-standing and formidable challenge in artificial intelligence. Recent breakthroughs, exemplified by large language models (LLMs) and…
RGSD: Rubric-Guided Self-Distillation
RubricHub-medical + HealthBench: +6.1ppMohammadHossein Rezaei, Anas Mahmoud +7
Rubrics have emerged as an alternative to RLVR in open-ended domains where a single ground-truth final answer is not available. Existing rubric-based training methods rely on an…
Reflexion: Language Agents with Verbal Reinforcement Learning
Pass@1: 91.0%Noah Shinn, Federico Cassano +4
Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains…
Kimi k1.5: Scaling Reinforcement Learning with LLMs
AIME 2024: 77.5%Kimi Team, Angang Du +94
Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement…
The Entropy Mechanism of RL for Reasoning LLMs
AIME24: 36.8%Ganqu Cui, Yuchen Zhang +15
This paper aims to overcome a major obstacle in scaling RL for reasoning with LLMs, namely the collapse of policy entropy. Such phenomenon is consistently observed across vast RL…
1-Shot RLVR: Reinforcement Learning for Reasoning with One Example
MATH500: 73.6%Yiping Wang, Qing Yang +12
We show that reinforcement learning with verifiable reward using one training example (1-shot RLVR) is effective in incentivizing the math reasoning capabilities of large…
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
AVG: 50.4%Andrew Zhao, Yiran Wu +9
Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from outcome-based…
Understanding R1-Zero-Like Training: A Critical Perspective
AIME 2024: 43.3%Zichen Liu, Changyu Chen +6
DeepSeek-R1-Zero has shown that reinforcement learning (RL) at scale can directly enhance the reasoning capabilities of LLMs without supervised fine-tuning. In this work, we…
DAPO: Demystifying Large-Scale LLM Reinforcement Learning
AIME 2024: 50.0%Qiying Yu, Zheng Zhang +33
Inference scaling empowers LLMs with unprecedented reasoning ability, with reinforcement learning as the core technique to elicit complex reasoning. However, key technical…
DeepSeekMath: Pushing the Limits of Mathematical Reasoning
MATH: 51.7%Zhihong Shao, Peiyi Wang +9
Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues…
Dota 2 with Large Scale Deep Reinforcement Learning — OpenAI Five
Win rate vs Public: 99.4%OpenAI, : +25
On April 13th, 2019, OpenAI Five became the first AI system to defeat the world champions at an esports game. The game of Dota 2 presents novel challenges for AI systems such as…
Decision Transformer: Reinforcement Learning via Sequence Modeling
DQN-Replay 1%: 267.5 ± 97.5Lili Chen, Kevin Lu +7
We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer…
Mastering Diverse Domains through World Models — DreamerV3
Atari 2600: 830%Danijar Hafner, Jurgis Pasukonis +2
Developing a general algorithm that learns to solve tasks across a wide range of applications has been a fundamental challenge in artificial intelligence. Although current…
World Models: Training Agents in Their Own Dreams
CarRacing-v0: 906 ± 21David Ha, Jürgen Schmidhuber
We explore building generative neural network models of popular reinforcement learning environments. Our world model can be trained quickly in an unsupervised manner to learn a…