MicroSpec: Accelerating Speculative Decoding with Lightweight In-Context Vocabularies 事件

MicroSpec: Accelerating Speculative Decoding with Lightweight In-Context Vocabularies arXiv:2605.26444v1 Announce Type: new Abstract: Large language models typically employ vocabularies of over 100k tokens, which creates a major computational bottleneck at the final linear projection layer when performing speculative decoding. Current methods for vocabulary pruning depend on either fixed or coarse-grained sub-vocabularies, requiring around 30k active tokens to preserve the quality of the draft