Skip to content
Sagan

Paper

MicroSpec: Accelerating Speculative Decoding with Lightweight In-Context Vocabularies

Unreadunread

AI summary

arXiv:2605. 26444v1 Announce Type: new Abstract: Large language models typically employ vocabularies of over 100k tokens, which creates a major computational bottleneck at the final linear projection layer when performing speculative decoding.