The authors address a limitation in RLHF: the reward model typically assumes a fixed "rationality" parameter (beta) that governs how consistently human preferences reflect true reward differences, but real human feedback is biased in context-dependent ways. They propose dynamically adjusting beta during reward learning using an LLM-as-judge to detect likely cognitive biases in each annotation, effectively downweighting comparisons that are probably unreliable or biased. Empirically, this produces a more rational downstream model even when training on heavily biased preference data.
Main takeaways:
- Standard RLHF uses a Boltzmann model with a fixed rationality parameter (beta) that assumes uniform annotator reliability, but human judgments are shaped by context-dependent cognitive biases
- The authors treat rationality as context- and annotation-dependent rather than fixed
- An LLM-as-judge assesses each preference comparison for likely cognitive bias, and beta is dynamically adjusted to downweight biased comparisons during reward learning
- This approach learns a more rational downstream model even when training on datasets with strongly biased preferences
- Addresses systematic deviations from reward-consistent behavior that arise contextually in human feedback