Abstract
Bimanual manipulation is a longstanding challenge in robotics due to thelarge number of degrees of freedom and the strict spatial and temporalsynchronization required to generate meaningful behavior. Humans learn bimanualmanipulation skills by watching other humans and by refining their abilitiesthrough play. In this work, we aim to enable robots to learn bimanualmanipulation behaviors from human video demonstrations and fine-tune themthrough interaction. Inspired by seminal work in psychology and biomechanics,we propose modeling the interaction between two hands as a serial kinematiclinkage -- as a screw motion, in particular, that we use to define a new actionspace for bimanual manipulation: screw actions. We introduce ScrewMimic, aframework that leverages this novel action representation to facilitatelearning from human demonstration and self-supervised policy fine-tuning. Ourexperiments demonstrate that ScrewMimic is able to learn several complexbimanual behaviors from a single human video demonstration, and that itoutperforms baselines that interpret demonstrations and fine-tune directly inthe original space of motion of both arms. For more information and videoresults, https://robin-lab.cs.utexas.edu/ScrewMimic/