Abstract
Robust Reinforcement Learning (RRL) is a promising Reinforcement Learning(RL) paradigm aimed at training robust to uncertainty or disturbances models,making them more efficient for real-world applications. Following thisparadigm, uncertainty or disturbances are interpreted as actions of a secondadversarial agent, and thus, the problem is reduced to seeking the agents'policies robust to any opponent's actions. This paper is the first to proposeconsidering the RRL problems within the positional differential game theory,which helps us to obtain theoretically justified intuition to develop acentralized Q-learning approach. Namely, we prove that under Isaacs's condition(sufficiently general for real-world dynamical systems), the same Q-functioncan be utilized as an approximate solution of both minimax and maximin Bellmanequations. Based on these results, we present the Isaacs Deep Q-Networkalgorithms and demonstrate their superiority compared to other baseline RRL andMulti-Agent RL algorithms in various environments.