Skip to content
Sagan

Paper

How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis

Unreadunread

AI summary

arXiv:2605. 24749v1 Announce Type: new Abstract: Reward modeling is not only a prediction problem: in KL-regularized policy optimization, the learned reward is exponentiated to define the deployed policy, so downstream value depends on errors in reward-tilted regions.