Language Modeling
cs.CL · 37 papers
LoRA: Low-Rank Adaptation of Large Language Models
WikiSQL: 74.0%Edward J. Hu, Yelong Shen +6
An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train…
Training Compute-Optimal Large Language Models
MMLU: 67.6%Jordan Hoffmann, Sebastian Borgeaud +20
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are…
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Language Modeling: 5xAlbert Gu, Tri Dao
Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module.…
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
GSM8K: 57.0%Jason Wei, Xuezhi Wang +7
We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex…
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Reddit TL;DR: 61.0%Rafael Rafailov, Archit Sharma +4
While large-scale unsupervised language models (LMs) learn broad world knowledge and some reasoning skills, achieving precise control of their behavior is difficult due to the…
Attention Is All You Need — Explaining the Transformer
BLEU: 28.4Ashish Vaswani, Noam Shazeer +6
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also…
Mem0: Scalable Long-Term Memory for AI Agents
LOCOMO: 67.13%Prateek Chhikara, Dev Khant +3
Large Language Models (LLMs) have demonstrated remarkable prowess in generating contextually coherent responses, yet their fixed context windows pose fundamental challenges for…
SkillOpt — Executive Strategy for Self-Evolving Agent Skills
best-or-tied evaluated cells: 52/52 cellsYifan Yang, Ziyang Gong +13
Agent skills today are hand-crafted, generated one-shot, or evolved through loosely controlled self-revision, none of which behaves like a deep-learning optimizer for the skill,…
RWKV: Reinventing RNNs for the Transformer Era
HellaSwag: 74.8%Bo Peng, Eric Alcaide +32
Transformers have revolutionized almost all natural language processing (NLP) tasks but suffer from memory and computational complexity that scales quadratically with sequence…
Tülu 3: Pushing Frontiers in Open Language Model Post-Training
GSM8K: 87.6%Nathan Lambert, Jacob Morrison +21
Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques…
Does RL Really Expand LLM Reasoning?
Omni-MATH-Train: 42.5%Yang Yue, Zhiqi Chen +6
Reinforcement Learning with Verifiable Rewards (RLVR) has recently demonstrated notable success in enhancing the reasoning performance of large language models (LLMs),…
Self-Instruct: Aligning LMs with Self-Generated Instructions
ROUGE-L: 39.9Yizhong Wang, Yeganeh Kordi +5
Large "instruction-tuned" language models (i.e., finetuned to respond to instructions) have demonstrated a remarkable ability to generalize zero-shot to new tasks. Nevertheless,…
GPQA: A Graduate-Level Google-Proof Q&A Benchmark
GPQA Main Set: 71.9%David Rein, Betty Li Hou +6
We present GPQA, a challenging dataset of 448 multiple-choice questions written by domain experts in biology, physics, and chemistry. We ensure that the questions are…
SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
SWE-bench: 1.96%Carlos E. Jimenez, John Yang +5
Language models have outpaced our ability to evaluate them effectively, but for their future development it is essential to study the frontier of their capabilities. We find…
Toolformer: Language Models Can Teach Themselves to Use Tools
ASDiv: 40.4%Timo Schick, Jane Dwivedi-Yu +6
Language models (LMs) exhibit remarkable abilities to solve new tasks from just a few examples or textual instructions, especially at scale. They also, paradoxically, struggle…
Constitutional AI: Harmlessness from AI Feedback
HHH Eval: 180Yuntao Bai, Saurav Kadavath +49
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through…
Scaling Instruction-Finetuned Language Models — Flan-PaLM & Flan-T5
MMLU: 75.2%Hyung Won Chung, Le Hou +33
Finetuning language models on a collection of datasets phrased as instructions has been shown to improve model performance and generalization to unseen tasks. In this paper we…
Weak-to-Strong Generalization: Eliciting Strong Capabilities with Weak Supervision
NLP Benchmarks: nearly 80%Collin Burns, Pavel Izmailov +10
Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior - for example, to evaluate…
The Era of 1-bit LLMs: BitNet b1.58
Energy Savings: 71.4xShuming Ma, Hongyu Wang +8
Recent research, such as BitNet, is paving the way for a new era of 1-bit Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant, namely BitNet b1.58, in…
Qwen3 Technical Report: Unified Reasoning & Dynamic Thinking Budgets
AIME'24: 85.7An Yang, Anfeng Li +58
In this work, we present Qwen3, the latest version of the Qwen model family. Qwen3 comprises a series of large language models (LLMs) designed to advance performance, efficiency,…
DeepSeek-V3 Technical Report
MMLU-Pro: 75.9%DeepSeek-AI, Aixin Liu +198
We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B total parameters with 37B activated for each token. To achieve efficient inference and…
Gemini 1.5: Unlocking Multimodal Long-Context
Synthetic Text Haystack: >99.7%Gemini Team, Petko Georgiev +1135
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning…
Gemini: A Family of Highly Capable Multimodal Models — Deep Dive
MMLU: 90.04%Gemini Team, Rohan Anil +1349
This report introduces a new family of multimodal models, Gemini, that exhibit remarkable capabilities across image, audio, video, and text understanding. The Gemini family…
PaLM: Scaling Language Modeling with Pathways
Model FLOPs Utilization (MFU): 46.2%Aakanksha Chowdhery, Sharan Narang +65
Large language models have been shown to achieve remarkable performance across a variety of natural language tasks using few-shot learning, which drastically reduces the number…