The authors extend inverse reinforcement learning (IRL) to interactive settings where the learner actively interacts with an expert rather than passively observing demonstrations. Traditional IRL just watches expert behavior and infers their reward function; interactive IRL (IIRL) has the learner trying to learn both the expert's reward function and a good policy for interacting with them simultaneously. They formulate this as a bi-level optimization problem and develop an algorithm (BISIRL) with formal convergence guarantees.
Main takeaways:
- Traditional IRL is passive (observe expert demonstrations); interactive IRL has the learner actively interact with the expert
- Formulated as bi-level optimization: lower level learns reward function, upper level learns interaction policy
- BISIRL algorithm solves this with inner loop (reward learning) and outer loop (policy learning)
- Formal convergence guarantees provided
- Validated through experiments on interactive scenarios