The authors revisit online (streaming) reinforcement learning—learning from a single pass through experience without replay buffers—focusing on how to make standard deep RL algorithms like DQN and C51 work in this setting. They find that C51 (a distributional RL algorithm) performs competitively with the recent StreamQ method, and argue that two properties matter: bounded gradients and variance-adjusted updates (like those in the Adam optimizer). Building on this, they derive an "Adaptive Q(λ)" algorithm using eligibility traces that achieves strong performance on Atari.
Main takeaways:
- Standard deep RL has relied on replay buffers for stability, but recent work (StreamQ) revisits purely online learning with eligibility traces and optimizer tweaks.
- The authors find that classic algorithms (DQN, C51) with the Adam optimizer already work well in the streaming setting, without needing StreamQ's special updates.
- Two key properties for robust online learning: bounded objective derivatives and variance-adjusted weight updates (Adam does the latter automatically).
- Their new Adaptive Q(λ) algorithm, combining eligibility traces with variance adjustment, achieves roughly double the human baseline on 55 Atari games, outperforming prior methods.