Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training 事件

Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training arXiv:2606.05610v1 Announce Type: new Abstract: The efficacy of continued pre-training for Large Language Models (LLMs) hinges upon hyperparameter configurations, such as learning rate and batch size. However, current practices often rely on heuristics or grid searches, leading to training instability and excessive costs. In this work, we first empirically discover that optimal hyperparameters follow stable and p