Abstract
The success of image generative models has enabled us to build methods thatcan edit images based on text or other user input. However, these methods arebespoke, imprecise, require additional information, or are limited to only 2Dimage edits. We present GeoDiffuser, a zero-shot optimization-based method thatunifies common 2D and 3D image-based object editing capabilities into a singlemethod. Our key insight is to view image editing operations as geometrictransformations. We show that these transformations can be directlyincorporated into the attention layers in diffusion models to implicitlyperform editing operations. Our training-free optimization method uses anobjective function that seeks to preserve object style but generate plausibleimages, for instance with accurate lighting and shadows. It also inpaintsdisoccluded parts of the image where the object was originally located. Given anatural image and user input, we segment the foreground object using SAM andestimate a corresponding transform which is used by our optimization approachfor editing. GeoDiffuser can perform common 2D and 3D edits like objecttranslation, 3D rotation, and removal. We present quantitative results,including a perceptual study, that shows how our approach is better thanexisting methods. Visit https://ivl.cs.brown.edu/research/geodiffuser.html formore information.