The authors introduce Auto-Rubric as Reward (ARR), a system that replaces opaque scalar reward models with explicit, human-interpretable evaluation criteria (rubrics) for aligning multimodal generative models. Instead of learning preference weights implicitly, ARR asks a vision-language model to externalize its preference knowledge as a structured checklist of quality dimensions before comparing outputs, then uses those rubrics to judge pairwise preferences. This makes evaluation more transparent and suppresses biases like positional preference; the structured feedback is then distilled into a binary reward via Rubric Policy Optimization (RPO) for stable policy training. On text-to-image and image-editing benchmarks, ARR-RPO beats both learned reward models and direct VLM judges.
Main takeaways:
- Rubrics externalize a VLM's implicit preference structure as explicit, inspectable quality dimensions before any pairwise comparison happens.
- This structured decomposition reduces evaluation biases (especially positional bias) and enables zero-shot or few-shot deployment with minimal supervision.
- Rubric Policy Optimization (RPO) converts multi-dimensional rubric scores into a robust binary reward signal for policy gradient training, avoiding scalar regression.
- The approach outperforms pairwise reward models and direct VLM judges on image generation and editing tasks, showing better data efficiency and reliability.