The authors present a system for speeding up LLM inference in real-time advertising (ad generation, targeting) using adaptive quantization, hierarchical sparsification, and prefix-tree parallel verification. They demonstrate significant speedup on two real-world advertising scenarios with acceptable quality loss, making LLMs practical for latency-sensitive commercial deployment.
Main takeaways:
- LLMs are too slow for real-time advertising systems without aggressive optimization
- Combined three techniques: adaptive group quantization (compress weights), layer-adaptive hierarchical sparsification (prune less important computations), and prefix-tree parallel verification (batch similar queries)
- Tested on two real advertising scenarios and achieved significant speedup with manageable quality degradation
- Makes LLMs operationally viable for commercial real-time deployment where millisecond latency matters