The authors use reinforcement learning (RL) on binary correctness rewards to improve a language model's ability to recall factual knowledge directly, without any chain-of-thought reasoning. Testing on closed-book QA, they get roughly 27% relative improvement across models. Mechanistically, RL doesn't teach new facts—it moves correct answers that already exist somewhere in the model's outputs (often in the low-probability tail) up into the top greedy predictions. The hardest training examples (where the right answer appeared in fewer than 1 in 128 pre-RL samples) drive 83% of the improvement, because even rare correct rollouts get reinforced.
Main takeaways:
- RL on simple correctness rewards improves factual recall by ~27% without reasoning chains or memorization of training data
- The mechanism is probability redistribution: moving existing correct answers from rare samples to greedy top-1 outputs
- The hardest examples (18% of training data) contribute 83% of the gain because their occasional correct rollouts get amplified
- RL acts as a tool for "unlocking" latent knowledge rather than installing new facts