Generalizability of the findings from a randomized controlled trial of a web-based substance use disorder intervention

Background and Objectives: There is growing concern regarding the generalizability of findings from randomized controlled trials (RCTs) of interventions for substance use disorders (SUDs). This study used a selection model approach to assess and improve the generalizability of an evaluation for a web-based SUD intervention by making the trial sample resemble the target population. Methods: The sample of the web-based SUD intervention (Therapeutic Education System vs. Treatment-as-usual; n = 507) was compared with the target population of SUD treatment-seeking individuals from the Treatment Episodes Data Set-Admissions (TEDS-A). Using weights based on the probabilities of RCT participation, we computed weighted treatment effects on retention and abstinence. Results: Substantial differences between the RCT sample and the target population was demonstrated in significant difference in the mean propensity scores (1.62 standard deviations at p <.001). The population effect on abstinence (12 weeks and 6 months) was statistically insignificant after weighting the data with the generalizability weight. Discussions and Conclusions: Generalizability of the findings from the RCT could be limited when the RCT sample does not well represent the target population. Scientific Significance: Application of generalizability weights can be a potentially useful tool to improve generalizability of RCT findings. (Am J Addict 2018;27:231–237).