This paper analyzes three quantization schemes for the key-value (KV) cache in transformers under a fixed bit budget: scalar quantization on both K and V (KV baseline), Walsh-Hadamard transform plus quantized Johnson-Lindenstrauss on V only (KQV), and QJL on both K and V (QKQV). The author derives statistical bounds showing that applying QJL to the key matrix K inflates inner-product variance, which softmax then amplifies nonlinearly. Empirically, at the most common bit budget (n=4), KQV consistently outperforms QKQV on KL divergence, geometric K error, and a 6D distance metric. At other budgets (n=2,3,5), QKQV sometimes wins on geometric K reconstruction but always loses on KL divergence, revealing a budget-dependent crossover and a K-V asymmetry.
Main takeaways:
- Three KV cache quantization schemes compared: KV (scalar MSE), KQV (WHT+MSE on K, WHT+MSE+QJL on V), QKQV (QJL on both K and V).
- At n=4 bits (the most practical setting), KQV beats QKQV on every metric—KL divergence, geometric K error, and 6D distance—across all tested distributions.
- The K-V asymmetry is unconditional: quantizing K with QJL consistently hurts KL divergence compared to leaving K as-is.
- A budget-dependent crossover exists: QKQV achieves better geometric K reconstruction at n ∈ {2,3,5}, KQV wins at n ∈ {4,6}.
- KL divergence bridges geometric K error to attention routing corruption and output collapse through softmax's Jensen amplification.