Soft-to-Hard Routing in Sparse Mixture-of-Experts Models 文章

arXiv:2605.02124v2 Announce Type: replace-cross Abstract: Softmax routing approaches hard top-1 routing as the temperature tends to zero, but the limiting passage is singular at router ties. This paper develops a boundary-layer calculus for this soft-to-hard limit in population squared-loss mixture-of-experts regression. For a router with logits $a_k(x;\phi)$, the relevant local quantity is the top-two margin $\Delta(x;\phi)$, and the relevant global quantity is the boundary mass $\mathbb{P}(\Delta(X;\phi)\le w)$. Under smoothness and transversality assumptions, coarea and tubular-neighborhood estimates show how this mass scales with the slab width; in the binary case the leading coefficient is an explicit surface integral over the routing interface.