Beyond Linearity in Attention Projections: The Case for Nonlinear Queries 事件

Beyond Linearity in Attention Projections: The Case for Nonlinear Queries arXiv:2603.13381v3 Announce Type: replace-cross Abstract: Recent algebraic analysis shows that in decoder-only and encoder-only transformers, the Query projection $W_Q$ may be set to identity without noticeable performance deterioration. This is possible because attention depends on $X$ only through the products $XW_Q, XW_K, XW_V$, allowing basis transformations to be absorbed by adjacent layers and propagated through the