Abstract
Aiming to replicate human-like dexterity, perceptual experiences, and motionpatterns, we explore learning from human demonstrations using a bimanual systemwith multifingered hands and visuotactile data. Two significant challengesexist: the lack of an affordable and accessible teleoperation system suitablefor a dual-arm setup with multifingered hands, and the scarcity ofmultifingered hand hardware equipped with touch sensing. To tackle the firstchallenge, we develop HATO, a low-cost hands-arms teleoperation system thatleverages off-the-shelf electronics, complemented with a software suite thatenables efficient data collection; the comprehensive software suite alsosupports multimodal data processing, scalable policy learning, and smoothpolicy deployment. To tackle the latter challenge, we introduce a novelhardware adaptation by repurposing two prosthetic hands equipped with touchsensors for research. Using visuotactile data collected from our system, welearn skills to complete long-horizon, high-precision tasks which are difficultto achieve without multifingered dexterity and touch feedback. Furthermore, weempirically investigate the effects of dataset size, sensing modality, andvisual input preprocessing on policy learning. Our results mark a promisingstep forward in bimanual multifingered manipulation from visuotactile data.Videos, code, and datasets can be found at https://toruowo.github.io/hato/ .