TY - JOUR
T1 - Impact Evaluation of Coronavirus Disease 2019 Policy
T2 - A Guide to Common Design Issues
AU - Haber, Noah A.
AU - Clarke-Deelder, Emma
AU - Salomon, Joshua A.
AU - Feller, Avi
AU - Stuart, Elizabeth A.
N1 - Publisher Copyright:
© 2021 The Author(s). Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health. All rights reserved.
PY - 2021/11/1
Y1 - 2021/11/1
N2 - Policy responses to coronavirus disease 2019 (COVID-19), particularly those related to nonpharmaceutical interventions, are unprecedented in scale and scope. However, evaluations of policy impacts require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and they differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate the strength of the evidence in COVID-19 health policy papers. Here we 1) introduce the basic suite of policy-impact evaluation designs for observational data, including cross-sectional analyses, pre-/post- A nalyses, interrupted time-series analysis, and difference-in-differences analysis; 2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19; and 3) provide decision-makers and reviewers with a conceptual and graphical guide to identifying these key violations. Our overall goal is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.
AB - Policy responses to coronavirus disease 2019 (COVID-19), particularly those related to nonpharmaceutical interventions, are unprecedented in scale and scope. However, evaluations of policy impacts require a complex combination of circumstance, study design, data, statistics, and analysis. Beyond the issues that are faced for any policy, evaluation of COVID-19 policies is complicated by additional challenges related to infectious disease dynamics and a multiplicity of interventions. The methods needed for policy-level impact evaluation are not often used or taught in epidemiology, and they differ in important ways that may not be obvious. Methodological complications of policy evaluations can make it difficult for decision-makers and researchers to synthesize and evaluate the strength of the evidence in COVID-19 health policy papers. Here we 1) introduce the basic suite of policy-impact evaluation designs for observational data, including cross-sectional analyses, pre-/post- A nalyses, interrupted time-series analysis, and difference-in-differences analysis; 2) demonstrate key ways in which the requirements and assumptions underlying these designs are often violated in the context of COVID-19; and 3) provide decision-makers and reviewers with a conceptual and graphical guide to identifying these key violations. Our overall goal is to help epidemiologists, policy-makers, journal editors, journalists, researchers, and other research consumers understand and weigh the strengths and limitations of evidence.
KW - coronavirus disease 2019
KW - difference-in-differences analysis
KW - impact evaluation
KW - interrupted time-series analysis
KW - nonpharmaceutical interventions
KW - policy
UR - http://www.scopus.com/inward/record.url?scp=85119876400&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85119876400&partnerID=8YFLogxK
U2 - 10.1093/aje/kwab185
DO - 10.1093/aje/kwab185
M3 - Article
C2 - 34180960
AN - SCOPUS:85119876400
SN - 0002-9262
VL - 190
SP - 2474
EP - 2486
JO - American journal of epidemiology
JF - American journal of epidemiology
IS - 11
ER -