What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science

Prasad Patil, Roger Peng, Jeffrey T Leek

Research output: Contribution to journalArticle

Abstract

A recent study of the replicability of key psychological findings is a major contribution toward understanding the human side of the scientific process. Despite the careful and nuanced analysis reported, the simple narrative disseminated by the mass, social, and scientific media was that in only 36% of the studies were the original results replicated. In the current study, however, we showed that 77% of the replication effect sizes reported were within a 95% prediction interval calculated using the original effect size. Our analysis suggests two critical issues in understanding replication of psychological studies. First, researchers’ intuitive expectations for what a replication should show do not always match with statistical estimates of replication. Second, when the results of original studies are very imprecise, they create wide prediction intervals—and a broad range of replication effects that are consistent with the original estimates. This may lead to effects that replicate successfully, in that replication results are consistent with statistical expectations, but do not provide much information about the size (or existence) of the true effect. In this light, the results of the Reproducibility Project: Psychology can be viewed as statistically consistent with what one might expect when performing a large-scale replication experiment.

Original languageEnglish (US)
Pages (from-to)539-544
Number of pages6
JournalPerspectives on Psychological Science
Volume11
Issue number4
DOIs
StatePublished - Jul 1 2016

Fingerprint

Research Personnel
Psychology
Reproducibility of Results

Keywords

  • p values
  • prediction intervals
  • replication
  • reproducibility
  • Reproducibility Project: Psychology

ASJC Scopus subject areas

  • Psychology(all)

Cite this

@article{d678154830204da4a3f0e32af49f06ca,
title = "What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science",
abstract = "A recent study of the replicability of key psychological findings is a major contribution toward understanding the human side of the scientific process. Despite the careful and nuanced analysis reported, the simple narrative disseminated by the mass, social, and scientific media was that in only 36{\%} of the studies were the original results replicated. In the current study, however, we showed that 77{\%} of the replication effect sizes reported were within a 95{\%} prediction interval calculated using the original effect size. Our analysis suggests two critical issues in understanding replication of psychological studies. First, researchers’ intuitive expectations for what a replication should show do not always match with statistical estimates of replication. Second, when the results of original studies are very imprecise, they create wide prediction intervals—and a broad range of replication effects that are consistent with the original estimates. This may lead to effects that replicate successfully, in that replication results are consistent with statistical expectations, but do not provide much information about the size (or existence) of the true effect. In this light, the results of the Reproducibility Project: Psychology can be viewed as statistically consistent with what one might expect when performing a large-scale replication experiment.",
keywords = "p values, prediction intervals, replication, reproducibility, Reproducibility Project: Psychology",
author = "Prasad Patil and Roger Peng and Leek, {Jeffrey T}",
year = "2016",
month = "7",
day = "1",
doi = "10.1177/1745691616646366",
language = "English (US)",
volume = "11",
pages = "539--544",
journal = "Perspectives on Psychological Science",
issn = "1745-6916",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - What Should Researchers Expect When They Replicate Studies? A Statistical View of Replicability in Psychological Science

AU - Patil, Prasad

AU - Peng, Roger

AU - Leek, Jeffrey T

PY - 2016/7/1

Y1 - 2016/7/1

N2 - A recent study of the replicability of key psychological findings is a major contribution toward understanding the human side of the scientific process. Despite the careful and nuanced analysis reported, the simple narrative disseminated by the mass, social, and scientific media was that in only 36% of the studies were the original results replicated. In the current study, however, we showed that 77% of the replication effect sizes reported were within a 95% prediction interval calculated using the original effect size. Our analysis suggests two critical issues in understanding replication of psychological studies. First, researchers’ intuitive expectations for what a replication should show do not always match with statistical estimates of replication. Second, when the results of original studies are very imprecise, they create wide prediction intervals—and a broad range of replication effects that are consistent with the original estimates. This may lead to effects that replicate successfully, in that replication results are consistent with statistical expectations, but do not provide much information about the size (or existence) of the true effect. In this light, the results of the Reproducibility Project: Psychology can be viewed as statistically consistent with what one might expect when performing a large-scale replication experiment.

AB - A recent study of the replicability of key psychological findings is a major contribution toward understanding the human side of the scientific process. Despite the careful and nuanced analysis reported, the simple narrative disseminated by the mass, social, and scientific media was that in only 36% of the studies were the original results replicated. In the current study, however, we showed that 77% of the replication effect sizes reported were within a 95% prediction interval calculated using the original effect size. Our analysis suggests two critical issues in understanding replication of psychological studies. First, researchers’ intuitive expectations for what a replication should show do not always match with statistical estimates of replication. Second, when the results of original studies are very imprecise, they create wide prediction intervals—and a broad range of replication effects that are consistent with the original estimates. This may lead to effects that replicate successfully, in that replication results are consistent with statistical expectations, but do not provide much information about the size (or existence) of the true effect. In this light, the results of the Reproducibility Project: Psychology can be viewed as statistically consistent with what one might expect when performing a large-scale replication experiment.

KW - p values

KW - prediction intervals

KW - replication

KW - reproducibility

KW - Reproducibility Project: Psychology

UR - http://www.scopus.com/inward/record.url?scp=84980042504&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84980042504&partnerID=8YFLogxK

U2 - 10.1177/1745691616646366

DO - 10.1177/1745691616646366

M3 - Article

C2 - 27474140

AN - SCOPUS:84980042504

VL - 11

SP - 539

EP - 544

JO - Perspectives on Psychological Science

JF - Perspectives on Psychological Science

SN - 1745-6916

IS - 4

ER -