When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift 事件

When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift arXiv:2605.25629v1 Announce Type: new Abstract: Weak-to-strong (W2S) generalization is a promising framework for scalable oversight, yet existing evaluations often test students under matched train--test distributions. Therefore, we study W2S preference learning under zero-shot distribution shift and find that strong students trained on weak preference labels can appear successful in-distribution whi