The author proposes QuIDE, a single metric for evaluating quantized neural networks that combines compression ratio, accuracy, and latency into an "Intelligence Index" score. Experiments across six settings (MNIST, CIFAR, ImageNet, and Llama-3-8B) show that the optimal quantization bit-width is task-dependent: 4-bit is best for simple tasks and large LLMs, while 8-bit is optimal for complex CNN tasks where 4-bit post-training quantization causes catastrophic accuracy collapse. The metric includes an accuracy-gated variant that flags non-viable configurations.
Main takeaways:
- QuIDE collapses the compression-accuracy-latency trade-off into a single Intelligence Index score: I = (C × P) / log₂(T+1), where C is compression, P is performance, and T is latency.
- 4-bit quantization is optimal for simple tasks (MNIST) and large LLMs (Llama-3-8B).
- 8-bit quantization is the sweet spot for complex CNN tasks like ResNet-18 on ImageNet, where 4-bit post-training quantization causes accuracy collapse.
- An accuracy-gated variant I' correctly flags non-viable configurations that the raw score would reward.
- The metric provides a ready-to-use fitness function for automated mixed-precision search.