The authors introduce TRIRL (Trust Region Inverse Reinforcement Learning), which tries to learn a reward function from expert demonstrations without either fully solving an RL problem at every iteration (like classical IRL) or suffering the instability of adversarial/discriminator-based methods. The key insight is that a trust-region-optimal policy for a large reward update is also globally optimal for a smaller update in the same direction, so you can do monotonic dual ascent using only local policy updates around the current policy. This bridges classical dual-ascent IRL (stable but expensive) and modern adversarial imitation learning (cheap but unstable), achieving 2.4x better performance than state-of-the-art and recovering reward functions that generalize to new dynamics.
Main takeaways:
- TRIRL avoids fully solving RL problems each iteration (expensive) and adversarial discriminator training (unstable) by doing explicit dual ascent with only local policy updates
- Key theoretical trick: a trust-region-optimal policy for a big reward change is globally optimal for a smaller change in the same direction, enabling monotonic improvement without global optimization
- Outperforms state-of-the-art imitation learning by 2.4x on aggregate inter-quartile mean across multiple tasks
- Learns reward functions in the traditional IRL sense—globally optimizable functions that match expert behavior, not just discriminator scores