The authors reframe Q-learning with linear function approximation as a switching dynamical system and analyze its convergence using the joint spectral radius (JSR), a tool from control theory that measures stability when a system switches between different modes. They show that whether Q-learning converges can be understood as whether the corresponding switched system is stable, and this framework applies to both deterministic updates and stochastic cases (i.i.d. observations or Markovian). The JSR perspective can be less conservative than traditional one-step norm bounds because it considers products of switching modes, and it also gives a new lens on regularized Q-learning.
Main takeaways:
- Q-learning with linear function approximation can be exactly modeled as a switched linear system, where convergence = stability in control-theory terms
- The joint spectral radius (JSR) captures how operator norms compound across multiple update steps, potentially giving tighter guarantees than single-step analysis
- The framework applies to deterministic updates, stochastic i.i.d. cases, and Markovian observation sequences
- Regularized Q-learning also fits naturally into this switched-system view, connecting projected Bellman equations to switched-system stability