The authors tackle reward modeling when human preferences are pluralistic (people disagree). Standard Bradley-Terry models can only capture disagreement by shrinking reward margins, and Gaussian reward models (which predict both mean and variance) are fundamentally non-identifiable from pairwise comparisons alone. They propose augmenting preference data with two coarse "anchor" labels per response (e.g., "good" vs. "bad") to resolve this non-identifiability, prove that two anchors are sufficient, and show both theoretically and empirically that the method improves reward modeling and downstream RLHF (PPO and best-of-N).
Main takeaways:
- Standard Bradley-Terry reward models can't properly represent disagreement—they just shrink reward gaps when preferences are noisy.
- Gaussian reward models (predicting mean and variance) are better in principle but suffer from non-identifiability: you can't uniquely recover the variance from pairwise preferences alone.
- Adding two anchor labels (coarse quality scores like "good" or "bad") per response resolves the identifiability problem.
- The authors prove that two anchors are mathematically sufficient, develop a joint training objective, and establish convergence rates for both mean and variance.
- Experiments on four diverging-preference datasets show consistent improvements in reward modeling, PPO training, and best-of-N selection.