The authors tested whether sharp, focused attention maps in vision-language models (VLMs) actually predict correct answers — spoiler, they don't. Using a "VLM Reliability Probe" on three model families (LLaVA-1.5, PaliGemma, Qwen2-VL), they found that attention structure has near-zero correlation with correctness (R ≈ 0.001), even though attention is causally necessary for the task. Instead, reliability lives in hidden-state geometry: late-layer logit margins and self-consistency (sampling multiple answers) are much stronger predictors. Causal ablation revealed an architectural split: late-fusion models (LLaVA) concentrate reliability in fragile bottleneck neurons, while early-fusion models (PaliGemma, Qwen2-VL) distribute it robustly across dimensions.
Main takeaways:
- Attention structure (sharpness, entropy) does not predict whether a VLM's answer is correct (R ≈ 0.001), debunking the "sharp attention = confidence" intuition.
- Hidden-state geometry — late-layer logit margins (R > 0.95 in two of three families) — and self-consistency (R = 0.43) are far stronger reliability indicators.
- Late-fusion architectures (LLaVA) are fragile: ablating top-5 probe neurons drops accuracy by 8.3 pp; early-fusion models (PaliGemma, Qwen2-VL) survive ablation of ~50% of hidden dimensions with ≤1 pp loss.
- Attention is causally necessary (masking top-30% patches drops accuracy 8–11 pp) but its structure doesn't reflect reliability — it's a permissive bottleneck, not an informative monitor.