Beyond Domain-Routed Supervision:
Cross-Teacher Guarding for MOPD

In traditional On-Policy Distillation, a static reward multiplier $\lambda$ scales all tokens uniformly:

$$J_\lambda(\theta) = \mathbb{E}_\theta \left[ \sum_{t=1}^T \left\{ \lambda \left( \ell^T_t - \ell^{\text{ref}}_t \right) - \left( \ell^\theta_t - \ell^{\text{ref}}_t \right) \right\} \right]$$

where $\ell^T_t, \ell^\theta_t, \ell^{\text{ref}}_t$ are log-probabilities of token $y_t$ given prefix $h_t$ under the teacher, student, and frozen reference. But reasoning rollouts contain a blend of universal structural markers (e.g. "Wait, let me double check") and hyper-specific domain preferences.

ROAD MOPD replaces the global scalar $\lambda$ with an agreement coefficient $\lambda_t$. For target teacher $T_r$ and auxiliary peer $T_a$, their token agreement relative to baseline $\pi_{\text{ref}}$ is:

$$s_t = \underbrace{\tanh\left( \frac{\ell^r_t - \ell^{\text{ref}}_t}{\tau} \right)}_{\text{routed-teacher evidence}} \times \underbrace{\tanh\left( \frac{\ell^a_t - \ell^{\text{ref}}_t}{\tau} \right)}_{\text{auxiliary-teacher evidence}}, \quad s_t \in [-1, 1]$$

The adaptive multiplier modulates the advantage:

$$\lambda_t = \text{clip}\left(1 + \rho_g s_t, \; \lambda_{\min}, \; \lambda_{\max}\right)$$

When both teachers favor the same change relative to baseline ($s_t > 0$), $\lambda_t > 1$ amplifies the learning signal. When they disagree ($s_t < 0$), $\lambda_t < 1$ attenuates supervision. Crucially, the auxiliary teacher is not treated as a domain solver, but as a compatibility sensor.

Token-level weighting filters isolated tokens, but gradient vectors aggregate over hundreds of tokens into the same weight matrices. If target-domain gradient $g_d = \nabla_\theta \mathcal{L}^{\text{agr}}_d$ points counter to the auxiliary gradient on that same trajectory $g'_d = \nabla_\theta \mathcal{L}'_{d \to a}$, parameter updates can harm other domains.

When $\langle g, g' \rangle < 0$, classic gradient surgery like PCGrad performs full orthogonal projection:

$$g_{\text{full}} = g - \frac{\langle g, g' \rangle}{\|g'\|_2^2} g'$$

The Problem with Full Projection: If $g$ and $g'$ are almost opposite ($\theta \to 180^\circ$), full projection practically erases the gradient ($\|g_{\text{full}}\| \to 0$), starving the target domain of learning capacity!

ROAD MOPD introduces an adaptive norm-retention constraint with retention floor $m \in (0, 1]$ (set to $m=0.6$ in experiments):

$$\tilde{g} = \arg\min_{v \in \text{conv}\{g, g_{\text{full}}\}} -\langle v, g' \rangle \quad \text{subject to } \|v\|_2^2 \ge m^2 \|g\|_2^2$$

The closed-form solution scales projection fraction $\alpha^* \in [0, 1]$:

$$\tilde{g} = g - \alpha^* \frac{\langle g, g' \rangle}{\|g'\|_2^2} g', \quad \alpha^* = \begin{cases} 0, & c \ge 0 \\ 1, & c < 0 \text{ and } \sqrt{1 - c^2} \ge m \\ 1 - \sqrt{1 - \frac{1 - m^2}{c^2}}, & c < 0 \text{ and } \sqrt{1 - c^2} < m \end{cases}$$

where $c = \frac{\langle g, g' \rangle}{\|g\|_2 \|g'\|_2} = \cos \theta$ is the gradient alignment. If full projection preserves at least $60\%$ of the target norm ($\sqrt{1-c^2} \ge 0.6$), it applies full conflict removal ($\alpha^*=1$). Otherwise, it projects as much as legally permitted without dropping below the $60\%$ norm floor.

What happens when scaling to $K > 2$ teachers (e.g. Math, Code, and Medical jointly)? ROAD MOPD scales naturally along both guards:

1. Multi-Teacher Credit Assignment: For a target domain $d$ with peer set $\mathcal{P}_d = \{j \mid j \ne d\}$, the agreement scores are averaged across all $K-1$ peers before gating: $$\bar{s}_{d,t} = \frac{1}{K-1} \sum_{j \in \mathcal{P}_d} \tanh\left(\frac{\ell^d_t - \ell^{\text{ref}}_t}{\tau}\right) \tanh\left(\frac{\ell^j_t - \ell^{\text{ref}}_t}{\tau}\right)$$
2. Sequential Projection under a Shared Norm Floor: Peers are ordered by their conflict severity (most negative cosine first). Crucially, the floor is preserved relative to the original target tensor $g^{(0)}_d$, not refreshed locally: $$\|g^{(q)}_d\|_2 \ge m \|g^{(0)}_d\|_2, \quad q = 1, \dots, K-1$$ This prevents cascading projections from silently whittling down the gradient norm to zero.