Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving 事件

Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving arXiv:2606.02964v1 Announce Type: cross Abstract: Large Language Model (LLM) inference relies on key-value (KV) caches to avoid redundant attention computation. While approximate KV cache retention techniques reduce memory usage by sacrificing model accuracy, lossless approaches instead evict KV cache blocks from GPU memory and reconstruct them on demand to preserve exact outputs. Existing lo