Estimates of External Validity Bias When Impact Evaluations Select Sites Nonrandomly

Stephen H. Bell, Robert B. Olsen, Larry L. Orr, Elizabeth A. Stuart

Research output: Contribution to journalArticlepeer-review

Abstract

Evaluations of educational programs or interventions are typically conducted in nonrandomly selected samples of schools or districts. Recent research has shown that nonrandom site selection can yield biased impact estimates. To estimate the external validity bias from nonrandom site selection, we combine lists of school districts that were selected nonrandomly for 11 educational impact studies with population data on student outcomes from the Reading First program. Our analysis finds that on average, if an impact study of Reading First were conducted in the districts from these 11 studies, the impact estimate would be biased downward. In particular, it would be 0.10 standard deviations lower than the impact in the broader population from which the samples were selected, a substantial bias based on several benchmarks of comparison.

Original languageEnglish (US)
Pages (from-to)318-335
Number of pages18
JournalEducational Evaluation and Policy Analysis
Volume38
Issue number2
DOIs
StatePublished - 2015

Keywords

  • convenience sampling
  • evaluation
  • external validity
  • generalizability
  • impacts
  • purposive sampling

ASJC Scopus subject areas

  • Education

Fingerprint

Dive into the research topics of 'Estimates of External Validity Bias When Impact Evaluations Select Sites Nonrandomly'. Together they form a unique fingerprint.

Cite this