Confidence-Adaptive SwiGLU for Mixture-of-Experts 事件

Confidence-Adaptive SwiGLU for Mixture-of-Experts arXiv:2606.00761v1 Announce Type: cross Abstract: SwiGLU has become a standard gated activation in modern Transformer MLPs, yet its gate sharpness -- the smoothness and selectivity of the gating function -- is typically fixed throughout training. In this work, we propose Confidence-Aware SwiGLU ($\kappa$-SwiGLU), a variant of SwiGLU for Mixture-of-Experts (MoE) models that adjusts expert gate sharpness according to token-level routing confidence