Abstract
We present LidarDM, a novel LiDAR generative model capable of producingrealistic, layout-aware, physically plausible, and temporally coherent LiDARvideos. LidarDM stands out with two unprecedented capabilities in LiDARgenerative modeling: (i) LiDAR generation guided by driving scenarios, offeringsignificant potential for autonomous driving simulations, and (ii) 4D LiDARpoint cloud generation, enabling the creation of realistic and temporallycoherent sequences. At the heart of our model is a novel integrated 4D worldgeneration framework. Specifically, we employ latent diffusion models togenerate the 3D scene, combine it with dynamic actors to form the underlying 4Dworld, and subsequently produce realistic sensory observations within thisvirtual environment. Our experiments indicate that our approach outperformscompeting algorithms in realism, temporal coherency, and layout consistency. Weadditionally show that LidarDM can be used as a generative world modelsimulator for training and testing perception models.