Quantizing the key-value cache to fewer bits saves memory, but current methods use the same bit-width for every attention head. Different heads have different importance, so a natural idea is to give more bits to important heads. The catch: each quantizer design has its own error-vs-bits curve, and if you fit the allocation using one quantizer's curve and apply it to another, the allocation can invert and perform worse than uniform quantization. RateQuant solves this by measuring each quantizer's actual distortion curve on a small calibration set, then uses rate-distortion theory (reverse waterfilling) to allocate bits optimally. On Qwen3-8B at 2.5 bits, it cuts perplexity from 49 to 15.
Main takeaways:
- Attention heads vary in importance, but naive mixed-precision can backfire
- Different quantizers have different error-decay rates (beta from 3.6 to 5.3)
- Using the wrong distortion model inverts bit allocation and hurts performance
- RateQuant calibrates each quantizer's curve and solves bit allocation with closed-form math
- Calibration takes 1.6 seconds on one GPU; inference has zero overhead
- 70% perplexity reduction on some quantizers at 2.5 average bits