Abstract
Randomized trials are considered the gold standard for making informeddecisions in medicine, yet they often lack generalizability to the patientpopulations in clinical practice. Observational studies, on the other hand,cover a broader patient population but are prone to various biases. Thus,before using an observational study for decision-making, it is crucial tobenchmark its treatment effect estimates against those derived from arandomized trial. We propose a novel strategy to benchmark observationalstudies beyond the average treatment effect. First, we design a statisticaltest for the null hypothesis that the treatment effects estimated from the twostudies, conditioned on a set of relevant features, differ up to sometolerance. We then estimate an asymptotically valid lower bound on the maximumbias strength for any subgroup in the observational study. Finally, we validateour benchmarking strategy in a real-world setting and show that it leads toconclusions that align with established medical knowledge.