AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates 事件

AdaDPO: Self-Adaptive Direct Preference Optimization with Balanced Gradient Updates arXiv:2605.28440v1 Announce Type: new Abstract: DPO has become a widely adopted alternative to RLHF for aligning LLMs with human preferences, eliminating the need for a separate reward model or RL loop. Recent theoretical analysis uncovers an asymmetric gradient behavior in DPO: the loss suppresses dispreferred responses substantially faster than it promotes preferred ones, causing the model to learn to avoid ba