Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes

Nianbo Dong, Wendy M. Reinke, Keith C. Herman, Catherine P. Bradshaw, Desiree W. Murray

Research output: Contribution to journalArticlepeer-review

Abstract

Background: There is a need for greater guidance regarding design parameters and empirical benchmarks for social and behavioral outcomes to inform assumptions in the design and interpretation of cluster randomized trials (CRTs). Objectives: We calculated the empirical reference values on critical research design parameters associated with statistical power for children’s social and behavioral outcomes, including effect sizes, intraclass correlations (ICCs), and proportions of variance explained by a covariate at different levels (R2). Subjects: Children from kindergarten to Grade 5 in the samples from four large CRTs evaluating the effectiveness of two classroom- and two school-level preventive interventions. Measures: Teacher ratings of students’ social and behavioral outcomes using the Teacher Observation of Classroom Adaptation–Checklist and the Social Competence Scale–Teacher. Research design: Two types of effect size benchmarks were calculated: (1) normative expectations for change and (2) policy-relevant demographic performance gaps. The ICCs and R2 were calculated using two-level hierarchical linear modeling (HLM), where students are nested within schools, and three-level HLM, where students were nested within classrooms, and classrooms were nested within schools. Results and Conclusions: Comprehensive tables of benchmarks and ICC values are provided to inform prevention researchers in interpreting the effect size of interventions and conduct power analyses for designing CRTs of children’s social and behavioral outcomes. The discussion also provides a demonstration for how to use the parameter reference values provided in this article to calculate the sample size for two- and three-level CRTs designs.

Original languageEnglish (US)
Pages (from-to)334-377
Number of pages44
JournalEvaluation Review
Volume40
Issue number4
DOIs
StatePublished - Aug 1 2016
Externally publishedYes

Keywords

  • cluster randomized trials (CRTs)
  • design parameters
  • effect size
  • empirical benchmarks
  • social and behavioral outcomes
  • statistical power

ASJC Scopus subject areas

  • Arts and Humanities (miscellaneous)
  • Social Sciences(all)

Fingerprint Dive into the research topics of 'Meaningful Effect Sizes, Intraclass Correlations, and Proportions of Variance Explained by Covariates for Planning Two- and Three-Level Cluster Randomized Trials of Social and Behavioral Outcomes'. Together they form a unique fingerprint.

Cite this