Scale AI · 2026 arXiv:2606.12507

Rubric-Guided Self-Distillation

Post-Training Without Rubric Verifiers

MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang, Utkarsh Tyagi, Advait Gosai, Razvan-Gabriel Dumitru, Aakash Sabharwal, Bing Liu, Yunzhong He

TL;DR

Training LLMs on open-ended tasks normally requires an expensive LLM judge to score each generated response against a rubric. RGSD eliminates the judge entirely: it uses the same base model, shown the rubric in its context, as a frozen teacher that provides dense per-token guidance to the unconditioned student. The result is quality comparable to judge-based GRPO at a fraction of the cost, with fewer hallucinated claims.

No LLM judge at training time 1 rollout/prompt vs. 16 for GRPO +6.1pp medical, +4.9pp science (avg) Fewer false claims than GRPO Dense per-token vs. sparse reward

Why This Matters

Modern LLM post-training has a clean story for math and coding: run the model, check the answer programmatically, update on the reward. But for medicine, science writing, legal reasoning, and creative tasks, there is no checker program. The community's answer has been rubricsChecklists of weighted criteria describing what an ideal response should contain. Each criterion is graded as met or not met.: structured checklists of criteria, graded by a capable LLM judge.

Rubric-based RLVRReinforcement Learning with Verifiable Rewards — using a verifier (here an LLM judge) to produce a scalar reward signal for policy gradient training. works, but it carries three hidden costs that compound at scale:

①Computational cost. Every generated rollout needs a grading call to a large LLM. With 16 rollouts per prompt and 12,519 medical training prompts, that's roughly 1 million judge calls per run.
②Verifier blind spots. The model learns to exploit the judge's idiosyncrasies — producing responses that score well on the training judge but not on a held-out panel. This is reward hacking.
③Credit assignment. A single end-of-trajectory scalar tells the model "this response was good/bad" but not which tokens made the difference across a long response.

RGSD attacks all three problems simultaneously by removing the judge from training entirely.

The Big Idea

If the base model already knows how to satisfy rubric criteria when it can see the rubric, why not teach the unconditioned model to behave as if it had seen the rubric — without actually showing it at inference time?

That is exactly what RGSD does. It instantiates two copies of the same base model: a student that sees only the prompt (as it will at inference time), and a frozen teacher that sees the prompt plus the rubric. At each training step, the student generates one response; the teacher — reading that same response token-by-token — tells the student what the next-token distribution should look like if it were rubric-aware. The student is trained to close this gap via a Jensen-Shannon divergenceA symmetric, bounded divergence between two probability distributions. RGSD uses a clipped version interpolating between forward and reverse KL. loss applied at every token position.

The result: dense, per-token supervision derived entirely from the model's own rubric-conditioned knowledge — no external judge, no multiple rollouts, no reward hacking surface.

How It Works

Training Instance Format

Each training example is a tuple $(q, R)$ where $q$ is a free-form prompt and $R = \{(c_i, w_i)\}_{i=1}^{K}$ is a rubric: a list of criteria $c_i$ with weights $w_i$. Given a response $y$, an LLM judge produces binary verdicts $v_i(q, c_i, y) \in \{0,1\}$ and the rubric score is:

$$s_J(q,R,y) = \frac{\sum_{i=1}^{K} w_i \cdot v_i(q,c_i,y)}{\sum_{i=1}^{K} w_i} \in [0,1]$$

Standard rubric-RL (GRPO) optimizes $\max_\theta \mathbb{E}_{(q,R)\sim\mathcal{D},\, y\sim\pi_\theta(\cdot|q)}[s_J(q,R,y)]$, which requires judge calls for every rollout. RGSD replaces this with a distillation objective.

The Self-Distillation Mechanism

RGSD creates two roles from a single base checkpoint $\theta_\text{base}$:

Student $\pi_S$ (trainable, $\theta_S \leftarrow \theta_\text{base}$): sees only the prompt $q$. Generates one on-policy rollout $y \sim \pi_S(\cdot|q)$.
Teacher $\pi_T$ (frozen, $\theta_T = \theta_\text{base}$): sees the prompt plus the rubric $R$. Given the student's prefix $y_{<t}$, produces a next-token distribution $\pi_T(\cdot | q, R, y_{<t})$.

The teacher's rubric context is wrapped in a template that ends with: "After understanding these evaluation criteria, provide your own thorough response to the problem. Address the criteria naturally without explicitly referencing them."

The RGSD Loss

The student is updated to match the teacher token-by-token under a clipped Jensen–Shannon divergence:

$$\mathcal{L}_\text{RGSD}(\theta_S) = \mathbb{E}_{(q,R)\sim\mathcal{D},\, y\sim\pi_S(\cdot|q)}\!\left[\frac{1}{T}\sum_{t=1}^{T} D_\beta^\text{clip}\!\left(\pi_S(\cdot|q,y_{<t})\,\Big\|\,\pi_T(\cdot|q,R,y_{<t})\right)\right]$$

where $D_\beta^\text{clip}$ interpolates between forward KL ($\beta=0$) and reverse KL ($\beta=1$). The paper uses $\beta=0.5$ throughout, with a clip threshold $\tau=0.05$.

Key property: every token in the rollout contributes a divergence term, providing $O(N)$ bits of supervision per episode versus the $O(1)$ bits of a trajectory-level scalar reward.

Handling Reasoning-Trace Models

For models like Qwen3-Thinking that produce <think>...</think> blocks before their final answer, the rubric-conditioned teacher's thinking trace routinely references rubric criteria by number — even when explicitly told not to. In 70–73% of medical teacher generations, the thinking trace contains explicit rubric references.

Distilling those tokens into a student that never sees the rubric would teach the student to hallucinate criterion references. The fix: mask the thinking-block tokens out of the distillation loss, applying it only to final-answer tokens. This yields a 2–4pp improvement on Qwen3-Thinking models.

Algorithm 1: RGSD

Require: Dataset $\mathcal{D}=\{(q_i,R_i)\}$; base model $\theta_\text{base}$; β; clip τ; epochs E

1: Initialize student $\theta_S \leftarrow \theta_\text{base}$; freeze teacher $\theta_T \leftarrow \theta_\text{base}$

2: for epoch = 1, …, E do

3: for each minibatch $\mathcal{B} \subset \mathcal{D}$ do

4: for each $(q, R) \in \mathcal{B}$ do

5: Sample rollout $y \sim \pi_S(\cdot | q)$ ← 1 rollout only

6: Compute $\pi_T(\cdot | q, R, y_{<t})$ for $t=1,\ldots,T$ ← frozen teacher

7: end for

8: Update $\theta_S$ by descending $\nabla_{\theta_S}\mathcal{L}_\text{RGSD}$ ← no judge call!

9: end for

10: end for

Results

RGSD was evaluated on four base models (Qwen-2.5-3B/7B-Instruct, Qwen3-4B/8B-Thinking) across medical and science domains. The evaluation judge was gpt-5.4 (different from the GRPO training judge gpt-4o-mini, preventing training-judge exploitation).

Length vs. quality. On Qwen-2.5 models, GRPO responses grow to 1.4–2.3× the length of RGSD responses by the final checkpoint, without a consistent score advantage. The factual-claim audit reveals why: GRPO's false-claim rate climbs from 30.5% to 45.1%, while RGSD's rises only to 35.1%. The extra tokens add unsupported claims, not better explanations.

Stronger judge ablation. Replacing gpt-4o-mini with gpt-oss-120b as the GRPO training judge improves GRPO substantially — it can outperform RGSD on science (39.2 vs. 35.9 on RubricHub-sci). But it still requires per-rollout verifier calls and produces longer responses (∼1,200 tokens vs. ∼700–900 for RGSD).

Enrichment ablation. Using a self-generated reference response as teacher context (OPSD-style) instead of the raw rubric yields 2–3pp lower rubric satisfaction. The rubric preserves all criteria simultaneously; a single sampled response may omit some.

Limitations & Open Questions

Requires a meaningful conditioning gap. RGSD transfers behavior the base model can already express when given the rubric. If the base model doesn't benefit from seeing the rubric (small gap), there's little signal to distill. The gap diagnostic (Table 1) is a cheap pre-training check.

Bounded by teacher quality. The teacher is the same base model — not a stronger external model. In domains where a substantially stronger verifier is available and affordable, judge-based GRPO can achieve higher rubric satisfaction than RGSD.

Evaluation still uses LLM judges. RGSD removes the judge from training, but rubric satisfaction at evaluation time is still measured by a held-out judge (gpt-5.4). Absolute scores inherit judge-specific biases.

Single-seed runs. Most main-table results are single-seed due to the cost of running GRPO at scale (∼1M judge calls per domain). A 3-seed envelope on the headline configuration shows tight variance (σ≈0.4pp), but small margins should be read as parity.

Reasoning-trace masking is a recipe choice. The thinking-token mask for Qwen3-Thinking is a practical fix for rubric leakage in the teacher's chain-of-thought. Standard instruction-tuned models don't need it.

Open questions: Can RGSD benefit from iterative teacher updates (co-training)? Does the approach scale to longer rubrics or more complex multi-turn tasks? Can the conditioning gap be improved by rubric augmentation?

Glossary

RLVR — Reinforcement Learning with Verifiable Rewards

Post-training paradigm where a model is trained via RL using rewards from a verifier. For math/code, the verifier is a program; for open-ended tasks, it's typically an LLM judge.

Rubric

A per-prompt checklist of weighted criteria $\{(c_i, w_i)\}$ describing what an ideal response should contain. Each criterion is graded binary (met/not met) by an LLM judge.

GRPO — Group Relative Policy Optimization

A policy gradient algorithm that samples G rollouts per prompt, scores each, and computes advantages relative to the group mean: $\hat{A}_i = (s_i - \mu)/\sigma$. Used as the baseline RL method in this paper.

On-Policy Distillation

Knowledge distillation where the student generates its own rollouts (on-policy), and the teacher provides next-token target distributions at each student-visited state. Provides O(N) bits of supervision vs. O(1) for scalar rewards.

Jensen-Shannon Divergence (JSD)

A symmetric, bounded ($[0, \ln 2]$) divergence between two distributions, defined as the average of the KL divergences to their mixture. RGSD uses a clipped variant parameterized by β ∈ [0,1] interpolating between forward KL (β=0) and reverse KL (β=1).

Reward Hacking

When a model learns to exploit the reward signal in ways that don't reflect true quality — e.g., producing verbose responses that satisfy a judge's surface-level patterns without being factually correct or genuinely helpful.

Rubric-Conditioning Gap

The difference in rubric satisfaction score between the base model given only the prompt vs. the same model given the prompt + rubric. This gap (30–46pp in the paper) represents the maximum signal RGSD can potentially transfer.

OPSD — On-Policy Self-Distillation

A prior method (Zhao et al., 2026) where teacher and student are the same model under different conditioning: teacher sees privileged info (reference response or ground truth), student sees only the question. RGSD extends this idea using rubrics as the privileged context.

Citation

@article{rezaei2026rgsd,
  title   = {Rubric-Guided Self-Distillation:
             Post-Training Without Rubric Verifiers},
  author  = {Rezaei, MohammadHossein and Mahmoud, Anas and
             Wang, Zihao and Tyagi, Utkarsh and Gosai, Advait and
             Dumitru, Razvan-Gabriel and Sabharwal, Aakash and
             Liu, Bing and He, Yunzhong},
  journal = {arXiv preprint arXiv:2606.12507},
  year    = {2026},
  url     = {https://arxiv.org/abs/2606.12507}
}