Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution 文章

arXiv:2505.24037v3 Announce Type: replace Abstract: Sparse large language models (LLMs) offer an attractive direction toward efficient deployment, but adapting them to downstream tasks remains challenging. The central difficulty is to enable effective task adaptation without sacrificing the efficiency advantages of sparsity. Existing fine-tuning methods are not well-suited to this setting, as they either introduce additional dense parameters or assume a fixed sparse topology, limiting their compatibility with sparse LLMs. In this paper, we propose Sparsity Evolution Fine-Tuning (SEFT), a fine-tuning framework designed specifically for sparse LLMs. SEFT allows sparse structure to evolve during fine-tuning by periodically reallocating sparse task-specific updates and reactivating previously pruned weights when beneficial. At the same time, SEFT preserves the efficiency advantages of sparsity through topology adaptation based on parameter importance.