← Flash PapersEat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-Based RL

Share this Flash Paper

Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-Based RL

https://flashpapers.ai/p/afbrqsrn32
Make your own
Hyun Ryu, Jong-Heon Kim +4 more
Details ▸
Win Rate vs Base Policy:68.0%· RubricHub-MedicalWin Rate vs Base Policy:74.1%· HealthBenchWin Rate vs Base Policy:61.5%· RubricHub-ScienceWin Rate vs Base Policy:67.1%· ResearchQARubric F1 Score:38.6%· RubricHub-MedicalResponse Length Reduction:2.6x shorter· RubricHub-Medical
Abstract

Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).

Concepts:GRPO