Abstract
Reconstructing models of the real world, including 3D geometry, appearance,and motion of real scenes, is essential for computer graphics and computervision. It enables the synthesizing of photorealistic novel views, useful forthe movie industry and AR/VR applications. It also facilitates the contentcreation necessary in computer games and AR/VR by avoiding laborious manualdesign processes. Further, such models are fundamental for intelligentcomputing systems that need to interpret real-world scenes and actions to actand interact safely with the human world. Notably, the world surrounding us isdynamic, and reconstructing models of dynamic, non-rigidly moving scenes is aseverely underconstrained and challenging problem. This state-of-the-art report(STAR) offers the reader a comprehensive summary of state-of-the-art techniqueswith monocular and multi-view inputs such as data from RGB and RGB-D sensors,among others, conveying an understanding of different approaches, theirpotential applications, and promising further research directions. The reportcovers 3D reconstruction of general non-rigid scenes and further addresses thetechniques for scene decomposition, editing and controlling, and generalizableand generative modeling. More specifically, we first review the common andfundamental concepts necessary to understand and navigate the field and thendiscuss the state-of-the-art techniques by reviewing recent approaches that usetraditional and machine-learning-based neural representations, including adiscussion on the newly enabled applications. The STAR is concluded with adiscussion of the remaining limitations and open challenges.