Soft-to-Hard Routing in Sparse Mixture-of-Experts Models 事件

Soft-to-Hard Routing in Sparse Mixture-of-Experts Models arXiv:2605.02124v2 Announce Type: replace-cross Abstract: Softmax routing approaches hard top-1 routing as the temperature tends to zero, but the limiting passage is singular at router ties. This paper develops a boundary-layer calculus for this soft-to-hard limit in population squared-loss mixture-of-experts regression. For a router with logits $a_k(x;\phi)$, the relevant local quantity is the top-two margin $\Delta(x;\phi)$, and the rel