When Data Is Scarce: Scaling Sparse Language Models with Repeated Training 事件

When Data Is Scarce: Scaling Sparse Language Models with Repeated Training arXiv:2606.01155v1 Announce Type: cross Abstract: Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained regimes where limited unique tokens require multi-epoch training. Our experiments span models up to 1.92B parameters in the fitting set, sparsity up to 93.75%, unique data budgets up to 2.6B toke