The authors study test-time scaling for personalized language generation: sample N outputs from a personalized policy model and pick the best with a personalized reward model. They prove that oracle selection (always picking the true-best candidate) yields logarithmic utility growth in N, but standard reward models fail to realize this. They derive a unified scaling law that decomposes any reward model's Best-of-N curve into four measurable terms and reveals two failure modes — user-level collapse (constant predictions for some users) and query-level reward hacking (negative correlation with quality for some queries). They then propose a probabilistic reward model that learns per-user, per-query variance to mitigate both failure modes, and show consistent test-time scaling across multiple policy models and personalized text tasks.
Main takeaways:
- Test-time personalization can scale by sampling many candidates and reranking, but only if the reward model is good enough.
- Oracle selection achieves log(N) utility growth; the authors prove this is the theoretical ceiling.
- A new scaling law decomposes Best-of-N performance into four quantities and identifies user-level collapse and query-level reward hacking as the key failure modes.
- A probabilistic reward model that outputs per-prediction variance successfully mitigates both failure modes.
- Experiments confirm the framework: test-time scaling works across policy models and personalized generation tasks when guided by the proposed reward model.