Abstract
Computer vision applications predict on digital images acquired by a camerafrom physical scenes through light. However, conventional robustness benchmarksrely on perturbations in digitized images, diverging from distribution shiftsoccurring in the image acquisition process. To bridge this gap, we introduce anew distribution shift dataset, ImageNet-ES, comprising variations inenvironmental and camera sensor factors by directly capturing 202k images witha real camera in a controllable testbed. With the new dataset, we evaluateout-of-distribution (OOD) detection and model robustness. We find that existingOOD detection methods do not cope with the covariate shifts in ImageNet-ES,implying that the definition and detection of OOD should be revisited toembrace real-world distribution shifts. We also observe that the model becomesmore robust in both ImageNet-C and -ES by learning environment and sensorvariations in addition to existing digital augmentations. Lastly, our resultssuggest that effective shift mitigation via camera sensor control cansignificantly improve performance without increasing model size. With thesefindings, our benchmark may aid future research on robustness, OOD, and camerasensor control for computer vision. Our code and dataset are available athttps://github.com/Edw2n/ImageNet-ES.