Abstract
Neural reconstruction approaches are rapidly emerging as the preferredrepresentation for 3D scenes, but their limited editability is still posing achallenge. In this work, we propose an approach for 3D scene inpainting -- thetask of coherently replacing parts of the reconstructed scene with desiredcontent. Scene inpainting is an inherently ill-posed task as there exist manysolutions that plausibly replace the missing content. A good inpainting methodshould therefore not only enable high-quality synthesis but also a high degreeof control. Based on this observation, we focus on enabling explicit controlover the inpainted content and leverage a reference image as an efficient meansto achieve this goal. Specifically, we introduce RefFusion, a novel 3Dinpainting method based on a multi-scale personalization of an image inpaintingdiffusion model to the given reference view. The personalization effectivelyadapts the prior distribution to the target scene, resulting in a lowervariance of score distillation objective and hence significantly sharperdetails. Our framework achieves state-of-the-art results for object removalwhile maintaining high controllability. We further demonstrate the generalityof our formulation on other downstream tasks such as object insertion, sceneoutpainting, and sparse view reconstruction.