Abstract
Visual control policies can encounter significant performance degradationwhen visual conditions like lighting or camera position differ from those seenduring training -- often exhibiting sharp declines in capability even for minordifferences. In this work, we examine robustness to a suite of these types ofvisual changes for RGB-D and point cloud based visual control policies. Toperform these experiments on both model-free and model-based reinforcementlearners, we introduce a novel Point Cloud World Model (PCWM) and point cloudbased control policies. Our experiments show that policies that explicitlyencode point clouds are significantly more robust than their RGB-Dcounterparts. Further, we find our proposed PCWM significantly outperformsprior works in terms of sample efficiency during training. Taken together,these results suggest reasoning about the 3D scene through point clouds canimprove performance, reduce learning time, and increase robustness for roboticlearners. Project Webpage: https://pvskand.github.io/projects/PCWM