Limitations of Normalization in Attention Mechanism 事件

Limitations of Normalization in Attention Mechanism arXiv:2508.17821v3 Announce Type: replace-cross Abstract: This paper investigates the limitations of the normalization in attention mechanisms. We begin with a theoretical framework that enables the identification of the model's selective ability and the geometric separation involved in token selection. Our analysis includes explicit bounds on distances and separation criteria for token vectors under softmax scaling. Through experiments with p