Abstract
Machine learning models traditionally assume that training and test data areindependently and identically distributed. However, in real-world applications,the test distribution often differs from training. This problem, known asout-of-distribution generalization, challenges conventional models. InvariantRisk Minimization (IRM) emerges as a solution, aiming to identify featuresinvariant across different environments to enhance out-of-distributionrobustness. However, IRM's complexity, particularly its bi-level optimization,has led to the development of various approximate methods. Our studyinvestigates these approximate IRM techniques, employing the ExpectedCalibration Error (ECE) as a key metric. ECE, which measures the reliability ofmodel prediction, serves as an indicator of whether models effectively captureenvironment-invariant features. Through a comparative analysis of datasets withdistributional shifts, we observe that Information Bottleneck-based IRM, whichcondenses representational information, achieves a balance in improving ECEwhile preserving accuracy relatively. This finding is pivotal, as itdemonstrates a feasible path to maintaining robustness without compromisingaccuracy. Nonetheless, our experiments also caution againstover-regularization, which can diminish accuracy. This underscores thenecessity for a systematic approach in evaluating out-of-distributiongeneralization metrics, one that beyond mere accuracy to address the nuancedinterplay between accuracy and calibration.