Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse 事件

Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse arXiv:2602.01203v3 Announce Type: replace Abstract: Large Language Models (LLMs) often assign disproportionate attention to the first token, a phenomenon known as the attention sink. Several recent approaches aim to address this issue, including Sink Attention in GPT-OSS and Gated Attention in Qwen3-Next. However, a comprehensive analysis of the relationship among these attention mechanisms is lac