The authors study model-based RL, where policies are trained on imagined rollouts from learned dynamics and reward models without querying the real environment. They analyze how errors in the learned models affect returns and derive the optimal sample allocation—how many samples to spend learning dynamics vs. rewards—under power-law scaling. They also show that zero-mean reward noise doesn't bias the REINFORCE gradient, only adds variance, which creates a tradeoff: is it better to get more rollouts with cheap noisy rewards, or fewer rollouts with expensive accurate rewards?
Main takeaways:
- Extends error-propagation analysis to MDPs with learned reward models and derives optimal dynamics-vs-reward sample allocation
- Identifies lower Lipschitz constants (smoother dynamics, reward, and policy) as a representation goal that tightens error bounds
- Shows zero-mean reward noise leaves REINFORCE gradient unbiased, only adding variance that decreases with more rollouts
- Frames the noisy-reward tradeoff as a one-dimensional optimization: more cheap noisy rollouts vs. fewer expensive accurate ones