Do Value Vectors in Deep Layers Need Context from the Residual Stream? 事件

Do Value Vectors in Deep Layers Need Context from the Residual Stream? arXiv:2606.02780v1 Announce Type: new Abstract: The success of the transformer architecture as the backbone of modern LLMs is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the residual stream as input and thereby produces context-dependent query, key, and value vectors. However, we find that model performance meaningfully improves when deeper layer