2014 Rossi Award Lecture: Beyond Internal Validity

Larry L. Orr

Research output: Contribution to journalArticle

Abstract

Background: For much of the last 40 years, the evaluation profession has been consumed in a battle over internal validity. Today, that battle has been decided. Random assignment, while still far from universal in practice, is almost universally acknowledged as the preferred method for impact evaluation. It is time for the profession to shift its attention to the remaining major flaws in the “standard model” of evaluation: (i) external validity and (ii) the high cost and low hit rate of experimental evaluations as currently practiced. Recommendations: To raise the profession’s attention to external validity, the author recommends some simple, easy steps to be taken in every evaluation. The author makes two recommendations to increase the number of interventions found to be effective within existing resources: First, a two-stage evaluation strategy in which a cheap, streamlined Stage 1 evaluation is followed by a more intensive Stage 2 evaluation only for those interventions found to be effective in a Stage 1 trial and, second, use of random assignment to guide the myriad program management decisions that must be made in the course of routine program operations. This article is not intended as a solution to these issues: It is intended to stimulate the evaluation community to take these issues more seriously and to develop innovative solutions.

Original languageEnglish (US)
Pages (from-to)167-178
Number of pages12
JournalEvaluation Review
Volume39
Issue number2
DOIs
StatePublished - Apr 11 2015

Keywords

  • content area
  • design and evaluation of programs and policies
  • methodological development
  • outcome evaluation (other than economic evaluation)

ASJC Scopus subject areas

  • Arts and Humanities (miscellaneous)
  • Social Sciences(all)

Fingerprint Dive into the research topics of '2014 Rossi Award Lecture: Beyond Internal Validity'. Together they form a unique fingerprint.

  • Cite this