TY - JOUR
T1 - Using Propensity Score Analysis of Survey Data to Estimate Population Average Treatment Effects
T2 - A Case Study Comparing Different Methods
AU - Dong, Nianbo
AU - Stuart, Elizabeth A.
AU - Lenis, David
AU - Quynh Nguyen, Trang
N1 - Funding Information:
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the National Institute of Mental Health (K25MH083846, R01MH099010, Principal Investigator Stuart) and the U.S. Department of Education, Institute of Education Sciences (R305D150001, Principal Investigators Stuart and Dong).
Publisher Copyright:
© The Author(s) 2020.
PY - 2020/2/1
Y1 - 2020/2/1
N2 - Background: Many studies in psychological and educational research aim to estimate population average treatment effects (PATE) using data from large complex survey samples, and many of these studies use propensity score methods. Recent advances have investigated how to incorporate survey weights with propensity score methods. However, to this point, that work had not been well summarized, and it was not clear how much difference the different PATE estimation methods would make empirically. Purpose: The purpose of this study is to systematically summarize the appropriate use of survey weights in propensity score analysis of complex survey data and use a case study to empirically compare the PATE estimates using multiple analysis methods that include ordinary least squares regression, weighted least squares regression, and various propensity score applications. Methods: We first summarize various propensity score methods that handle survey weights. We then demonstrate the performance of various analysis methods using a nationally representative data set, the Early Childhood Longitudinal Study–Kindergarten to estimate the effects of preschool on children’s academic achievement. The correspondence of the results was evaluated using multiple criteria. Results and Conclusions: It is important for researchers to think carefully about their estimand of interest and use methods appropriate for that estimand. If interest is in drawing inferences to the survey target population, it is important to take the survey weights into account, particularly in the outcome analysis stage for estimating the PATE. The case study shows, however, not much difference among various analysis methods in one applied example.
AB - Background: Many studies in psychological and educational research aim to estimate population average treatment effects (PATE) using data from large complex survey samples, and many of these studies use propensity score methods. Recent advances have investigated how to incorporate survey weights with propensity score methods. However, to this point, that work had not been well summarized, and it was not clear how much difference the different PATE estimation methods would make empirically. Purpose: The purpose of this study is to systematically summarize the appropriate use of survey weights in propensity score analysis of complex survey data and use a case study to empirically compare the PATE estimates using multiple analysis methods that include ordinary least squares regression, weighted least squares regression, and various propensity score applications. Methods: We first summarize various propensity score methods that handle survey weights. We then demonstrate the performance of various analysis methods using a nationally representative data set, the Early Childhood Longitudinal Study–Kindergarten to estimate the effects of preschool on children’s academic achievement. The correspondence of the results was evaluated using multiple criteria. Results and Conclusions: It is important for researchers to think carefully about their estimand of interest and use methods appropriate for that estimand. If interest is in drawing inferences to the survey target population, it is important to take the survey weights into account, particularly in the outcome analysis stage for estimating the PATE. The case study shows, however, not much difference among various analysis methods in one applied example.
KW - complex surveys
KW - equivalence test
KW - population average treatment effects
KW - propensity scores
KW - survey weights
UR - http://www.scopus.com/inward/record.url?scp=85088145315&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85088145315&partnerID=8YFLogxK
U2 - 10.1177/0193841X20938497
DO - 10.1177/0193841X20938497
M3 - Article
C2 - 32672113
AN - SCOPUS:85088145315
SN - 0193-841X
VL - 44
SP - 84
EP - 108
JO - Evaluation Review
JF - Evaluation Review
IS - 1
ER -