Generalizability of Risk Models for Treatment-Resistant Depression Across Three Health Systems.

Predicting which people with major depressive disorder will develop treatment-resistant depression from electronic health record data alone is not yet accurate enough for clinical use. Models built using sociodemographic features, medications, and diagnosis codes from three large U.S. health systems performed only slightly better than chance, with discrimination scores (AUROC) between 0.58 and 0.64 within the same health system and dropping to 0.51–0.58 when applied to a different system. Agreement between models from different sites was low, with concordance correlation coefficients ranging from 0.13 to 0.38. Age and sex were the most important features driving differences in model predictions. The findings suggest that readily available electronic health record data are insufficient for reliable prediction and that further improvement will require additional data types.

Study at a glance

Abstract

As multiple strategies have emerged for managing treatment-resistant major depressive disorder, efficient identification of individuals at elevated risk for this outcome earlier in their illness course remains essential. We extracted electronic health records data for all individuals with a diagnosis of major depressive disorder who received an index antidepressant prescription in the clinical networks of three geographically-distinct health systems - Mass General-Brigham (MGB), Vanderbilt University Medical Center (VUMC), and Geisinger Clinic (GC) - between April 1, 2004, and March 30, 2022. The primary outcome, treatment resistant depression, was defined as provision of electroconvulsive therapy, transcranial magnetic stimulation, vagus nerve stimulation, prescription of either ketamine or esketamine or monoamine oxidase inhibitors (MAOIs), or failed trials of more than two antidepressants. We applied L1-regularized regression to sociodemographic features, medications, and ICD10 diagnostic code counts to fit a model of treatment resistance in each of the three cohorts. For each, we then estimated generalizable model performance, aka external validity, across the other two cohorts. Model concordance was measured with Concordance Correlation Coefficients (CCCs) and random forest regression analyses were used to estimate importance of features predicting discordance. Across sites, discrimination performance ranged from Area Under the Receiver Operating Characteristic curves (AUROCs) 0.58 - 0.64 on internal validation and 0.51 - 0.58 on external validation. Area Under the Precision-Recall curve (AUPRC) ranged from 0.1-0.13 on internal validation and averaged 0.07-0.13 in external validation on the same test sets held out at each site. On the same testing set, CCCs were 0.13 for the VUMC MGB models, 0.18 for VUMCGC models, and 0.38 for MGB GC models. These results indicate the MGB and GC models were better correlated, but none were well correlated. Important features predicting discordance were dominated primarily by age and secondarily coded sex. These linear models demonstrated consistent aggregate performance and discordant individual performance across three, disparate major health systems. The inclusion of large and heterogeneous samples suggest that further improvement may require incorporation of data types beyond those readily available in EHR. Close attention to performance by key subgroups is indicated to ensure models do not perform disparately or unfairly. Prospective studies to evaluate the extent to which clinical models might improve early identification and outcomes are warranted.