Skip to content
Sagan

Paper

Service-Induced Congestion in Memory-Constrained LLM Serving

Unreadunread

AI summary

arXiv:2606. 15555v1 Announce Type: cross Abstract: In large language model (LLM) serving, each request accumulates persistent graphics processing unit (GPU) memory during service as its key-value cache grows with every generated token.