Weight Decay Improves Language Model Plasticity 事件

Weight Decay Improves Language Model Plasticity arXiv:2602.11137v2 Announce Type: replace-cross Abstract: Large language models are typically trained in two broad phases: pretraining to produce a base model, followed by further training to improve downstream performance. However, hyperparameter optimization and scaling laws are studied primarily from the perspective of the base model's validation loss, overlooking a crucial model property: downstream adaptability. In this work, we study pretrai