Towards the Connection between Activation Sparsity and Flat Minima 事件

Towards the Connection between Activation Sparsity and Flat Minima arXiv:2605.25612v1 Announce Type: cross Abstract: The observation that activation sparsity emerges in MLP blocks of standardly trained Transformers offers an opportunity to drastically reduce computation costs without sacrificing performance. To theoretically explain this phenomenon, existing works have shown that activation sparsity does not result from the data properties or data fitting but from the implicit bias of the train