The authors tackle reinforcement learning on long tasks with sparse rewards by letting an RL agent choose how far ahead to plan at each step. Instead of always executing fixed-length action sequences ("chunks"), their ACSAC system uses a Transformer-based critic to score different chunk sizes and picks whichever looks most promising in the current state. They prove this adaptive scheme still converges and show it beats fixed-chunk baselines on robotic manipulation benchmarks.
Main takeaways:
- Standard actor-critic methods struggle on long-horizon tasks because errors compound over many time steps.
- Action chunking (executing multi-step plans) helps, but a fixed chunk size forces a trade-off: large chunks ignore new information, small chunks produce jerky behavior.
- ACSAC evaluates chunks of different lengths with a causal Transformer and adaptively picks the best size at each decision point.
- Experiments on OGBench manipulation tasks show state-of-the-art results in both offline RL and offline-to-online settings.
- The authors provide a contraction-mapping proof that the adaptive Bellman operator has a unique fixed point.