Vision-language models often "hallucinate" objects that aren't in the image because they rely too heavily on language patterns and under-weight visual information. The authors introduce Positive-and-Negative Decoding (PND), a training-free method that intervenes during text generation. It runs two parallel paths: one that amplifies visual signals and one that constructs counterfactuals to suppress language-prior-driven outputs. By contrasting these paths at each token, PND forces the model to stick to what's actually in the image, achieving state-of-the-art hallucination reduction.
Main takeaways:
- Vision-language models under-weight visual features relative to language priors during generation
- PND is training-free — it only changes how tokens are selected during inference
- Dual paths: positive path boosts visual evidence, negative path penalizes prior-driven guesses
- Achieves best results on POPE, MME, and CHAIR hallucination benchmarks
- The core insight is attention imbalance: visual attention scores are too low