TY - JOUR
T1 - Using propensity scores in difference-in-differences models to estimate the effects of a policy change
AU - Stuart, Elizabeth A.
AU - Huskamp, Haiden A.
AU - Duckworth, Kenneth
AU - Simmons, Jeffrey
AU - Song, Zirui
AU - Chernew, Michael E.
AU - Barry, Colleen L.
N1 - Funding Information:
Acknowledgements We gratefully acknowledge funding support from the Commonwealth Fund [Grant # 20130499]. Dr. Stuart’s time was partially supported by the National Institute of Mental Health
Publisher Copyright:
© 2014, Springer Science+Business Media New York.
PY - 2014/11/18
Y1 - 2014/11/18
N2 - Difference-in-difference (DD) methods are a common strategy for evaluating the effects of policies or programs that are instituted at a particular point in time, such as the implementation of a new law. The DD method compares changes over time in a group unaffected by the policy intervention to the changes over time in a group affected by the policy intervention, and attributes the “difference-in-differences” to the effect of the policy. DD methods provide unbiased effect estimates if the trend over time would have been the same between the intervention and comparison groups in the absence of the intervention. However, a concern with DD models is that the program and intervention groups may differ in ways that would affect their trends over time, or their compositions may change over time. Propensity score methods are commonly used to handle this type of confounding in other non-experimental studies, but the particular considerations when using them in the context of a DD model have not been well investigated. In this paper, we describe the use of propensity scores in conjunction with DD models, in particular investigating a propensity score weighting strategy that weights the four groups (defined by time and intervention status) to be balanced on a set of characteristics. We discuss the conceptual issues associated with this approach, including the need for caution when selecting variables to include in the propensity score model, particularly given the multiple time point nature of the analysis. We illustrate the ideas and method with an application estimating the effects of a new payment and delivery system innovation (an accountable care organization model called the “Alternative Quality Contract” (AQC) implemented by Blue Cross Blue Shield of Massachusetts) on health plan enrollee out-of-pocket mental health service expenditures. We find no evidence that the AQC affected out-of-pocket mental health service expenditures of enrollees.
AB - Difference-in-difference (DD) methods are a common strategy for evaluating the effects of policies or programs that are instituted at a particular point in time, such as the implementation of a new law. The DD method compares changes over time in a group unaffected by the policy intervention to the changes over time in a group affected by the policy intervention, and attributes the “difference-in-differences” to the effect of the policy. DD methods provide unbiased effect estimates if the trend over time would have been the same between the intervention and comparison groups in the absence of the intervention. However, a concern with DD models is that the program and intervention groups may differ in ways that would affect their trends over time, or their compositions may change over time. Propensity score methods are commonly used to handle this type of confounding in other non-experimental studies, but the particular considerations when using them in the context of a DD model have not been well investigated. In this paper, we describe the use of propensity scores in conjunction with DD models, in particular investigating a propensity score weighting strategy that weights the four groups (defined by time and intervention status) to be balanced on a set of characteristics. We discuss the conceptual issues associated with this approach, including the need for caution when selecting variables to include in the propensity score model, particularly given the multiple time point nature of the analysis. We illustrate the ideas and method with an application estimating the effects of a new payment and delivery system innovation (an accountable care organization model called the “Alternative Quality Contract” (AQC) implemented by Blue Cross Blue Shield of Massachusetts) on health plan enrollee out-of-pocket mental health service expenditures. We find no evidence that the AQC affected out-of-pocket mental health service expenditures of enrollees.
KW - Causal inference
KW - Mental health spending
KW - Natural experiment
KW - Non-experimental study
KW - Policy evaluation
UR - http://www.scopus.com/inward/record.url?scp=84911990800&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84911990800&partnerID=8YFLogxK
U2 - 10.1007/s10742-014-0123-z
DO - 10.1007/s10742-014-0123-z
M3 - Article
C2 - 25530705
AN - SCOPUS:84911990800
SN - 1387-3741
VL - 14
SP - 166
EP - 182
JO - Health Services and Outcomes Research Methodology
JF - Health Services and Outcomes Research Methodology
IS - 4
ER -