Future-KL Regularized GRPO: Process-Level Credit Assignment from $f$-Divergence Regularization 事件

Future-KL Regularized GRPO: Process-Level Credit Assignment from $f$-Divergence Regularization arXiv:2601.10201v2 Announce Type: replace-cross Abstract: Group Relative Policy Optimization (GRPO) is widely used for critic-free Large Language Model (LLM) post-training, but its KL regularization is usually implemented as a local loss-side token penalty. We show that this misses the policy-gradient signal induced by autoregressive KL regularization. Unlike standard KL-regularized Reinforcement Lear