The authors tested three pruning methods (random, magnitude-based, and activation-aware "Wanda") on three instruction-tuned models at various sparsity levels, measuring both perplexity and bias on a benchmark. They found a paradox: the smartest pruning method (Wanda) preserves perplexity nearly perfectly but amplifies bias the most—at 70% sparsity, 47-59% of previously unbiased outputs became stereotypical. Random pruning destroys language ability but produces only random-chance bias. They also show unstructured pruning gives zero actual speedup or storage savings on real edge hardware, and that perplexity doesn't predict behavioral changes.
Main takeaways:
- "Smart" activation-aware pruning (Wanda) preserves perplexity but amplifies bias far more than simpler methods
- At 70% sparsity, stereotype reliance increases 83.7% and half of unbiased items flip to biased
- Random pruning destroys capability (perplexity reaching 10^8) but produces only random-chance bias
- Unstructured pruning provides zero storage or latency savings on real edge hardware despite being the primary motivation
- Perplexity-based evaluation gives false assurance of behavioral equivalence; pruning causes 3x more bias flips than quantization