Abstract
Various methods for Multi-Agent Reinforcement Learning (MARL) have beendeveloped with the assumption that agents' policies are based on accurate stateinformation. However, policies learned through Deep Reinforcement Learning(DRL) are susceptible to adversarial state perturbation attacks. In this work,we propose a State-Adversarial Markov Game (SAMG) and make the first attempt toinvestigate different solution concepts of MARL under state uncertainties. Ouranalysis shows that the commonly used solution concepts of optimal agent policyand robust Nash equilibrium do not always exist in SAMGs. To circumvent thisdifficulty, we consider a new solution concept called robust agent policy,where agents aim to maximize the worst-case expected state value. We prove theexistence of robust agent policy for finite state and finite action SAMGs.Additionally, we propose a Robust Multi-Agent Adversarial Actor-Critic (RMA3C)algorithm to learn robust policies for MARL agents under state uncertainties.Our experiments demonstrate that our algorithm outperforms existing methodswhen faced with state perturbations and greatly improves the robustness of MARLpolicies. Our code is public onhttps://songyanghan.github.io/what_is_solution/.