One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models 事件

One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models arXiv:2603.03291v2 Announce Type: replace Abstract: Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences. However, RM-based preference-tuning is vulnerable to reward hacking, whereby LM policies learn undesirable behaviors from flawed RMs. By systematically measuring biases in five high-quality RMs, including the state-of-the-art, we find that issues