The author found that putting numeric labels (like "quality score: 3") directly on images systematically biases vision-language models' quality judgments—this "anchoring bias" is 2.5× stronger than the effect of severely degrading the actual image quality. Layer-by-layer probing of the model's internal representations reveals a dissociation: the layers where the model first understands the anchor number (early-to-mid layers) are not the same layers that best predict actual quality (deeper layers). Some architectures fuse vision and language immediately at layer 1–2, while others show partial or no fusion.
Main takeaways:
- Numeric anchors on images bias VLM quality judgments far more than actual image quality changes—it's a robust, causal effect across six models.
- The layers where the model "reads" the anchor saturate early (layers 12–34), but the best layers for predicting real quality are deeper.
- This dissociation suggests the model processes the anchor separately from the actual visual quality signal.
- Fusion timing (when vision and language combine) varies by architecture: some fuse instantly, others never fully integrate.
- Visual anchoring bias isn't reducible to the anchor changing what the image looks like—it's a true cognitive-style bias in the model.