Reliability concerns in the repeated computerized assessment of attention in children

Theodore A Zabel, Christian Von Thomsen, Carolyn Cole, Rebecca Martin, Ernest M Mahone

Research output: Contribution to journalArticle

Abstract

Assessment of attentional processes via computerized assessment is frequently used to quantify intra-individual cognitive improvement or decline in response to treatment. However, assessment of intra-individual change is highly dependent on sufficient test reliability. We examined the test-retest reliability of selected variables from one popular computerized continuous performance test (CPT)i.e., the Conners' CPT - Second Edition (CPT-II). Participants were 39 healthy children (20 girls) ages 6-18 without intellectual impairment (mean PPVT-III SS = 102.6), LD, or psychiatric disorders (DICA-IV). Test-retest reliability over the 3-8 month interval (mean = 6 months) was acceptable (Intraclass Correlations [ICC] =.82 to.92) on comparison measures (Beery Test of Visual Perception, WISC-IV Block Design, PPVT-III). In contrast, test-retest reliability was only modest for CPT-II raw scores (ICCs ranging from.62 to.82) and T-scores (ICCs ranging from.33 to.65) for variables of interest (Omissions, Commissions, Variability, Hit Reaction Time, and Attentiveness). Using test-retest reliability information published in the CPT-II manual, 90% confidence intervals based on reliable change index (RCI) methodology were constructed to examine the significance of test-retest difference/change scores. Of the participants in this sample of typically developing youth, 30% generated intra-individual changes in T-scores on the Omissions and Attentiveness variables that exceeded the 90% confidence intervals and qualified as statistically rare changes in score. These results suggest a considerable degree of normal variability in CPT-II test scores over extended test-retest intervals, and suggest a need for caution when interpreting test score changes in neurologically unstable clinical populations.

Original languageEnglish (US)
Pages (from-to)1213-1231
Number of pages19
JournalClinical Neuropsychologist
Volume23
Issue number7
DOIs
StatePublished - Sep 2009

Fingerprint

Reproducibility of Results
Confidence Intervals
Visual Perception
Process Assessment (Health Care)
Reaction Time
Psychiatry
Performance Test
Population
Test-retest Reliability
Therapeutics

Keywords

  • Attention
  • Conners' Continuous Performance Test
  • Neuropsychology
  • Normal development
  • Reliable Change Index
  • Serial assessment

ASJC Scopus subject areas

  • Psychiatry and Mental health
  • Neuropsychology and Physiological Psychology
  • Clinical Psychology

Cite this

Reliability concerns in the repeated computerized assessment of attention in children. / Zabel, Theodore A; Von Thomsen, Christian; Cole, Carolyn; Martin, Rebecca; Mahone, Ernest M.

In: Clinical Neuropsychologist, Vol. 23, No. 7, 09.2009, p. 1213-1231.

Research output: Contribution to journalArticle

Zabel, Theodore A ; Von Thomsen, Christian ; Cole, Carolyn ; Martin, Rebecca ; Mahone, Ernest M. / Reliability concerns in the repeated computerized assessment of attention in children. In: Clinical Neuropsychologist. 2009 ; Vol. 23, No. 7. pp. 1213-1231.
@article{eb89bd94bfba4f62aae59fa892318308,
title = "Reliability concerns in the repeated computerized assessment of attention in children",
abstract = "Assessment of attentional processes via computerized assessment is frequently used to quantify intra-individual cognitive improvement or decline in response to treatment. However, assessment of intra-individual change is highly dependent on sufficient test reliability. We examined the test-retest reliability of selected variables from one popular computerized continuous performance test (CPT)i.e., the Conners' CPT - Second Edition (CPT-II). Participants were 39 healthy children (20 girls) ages 6-18 without intellectual impairment (mean PPVT-III SS = 102.6), LD, or psychiatric disorders (DICA-IV). Test-retest reliability over the 3-8 month interval (mean = 6 months) was acceptable (Intraclass Correlations [ICC] =.82 to.92) on comparison measures (Beery Test of Visual Perception, WISC-IV Block Design, PPVT-III). In contrast, test-retest reliability was only modest for CPT-II raw scores (ICCs ranging from.62 to.82) and T-scores (ICCs ranging from.33 to.65) for variables of interest (Omissions, Commissions, Variability, Hit Reaction Time, and Attentiveness). Using test-retest reliability information published in the CPT-II manual, 90{\%} confidence intervals based on reliable change index (RCI) methodology were constructed to examine the significance of test-retest difference/change scores. Of the participants in this sample of typically developing youth, 30{\%} generated intra-individual changes in T-scores on the Omissions and Attentiveness variables that exceeded the 90{\%} confidence intervals and qualified as statistically rare changes in score. These results suggest a considerable degree of normal variability in CPT-II test scores over extended test-retest intervals, and suggest a need for caution when interpreting test score changes in neurologically unstable clinical populations.",
keywords = "Attention, Conners' Continuous Performance Test, Neuropsychology, Normal development, Reliable Change Index, Serial assessment",
author = "Zabel, {Theodore A} and {Von Thomsen}, Christian and Carolyn Cole and Rebecca Martin and Mahone, {Ernest M}",
year = "2009",
month = "9",
doi = "10.1080/13854040902855358",
language = "English (US)",
volume = "23",
pages = "1213--1231",
journal = "Clinical Neuropsychologist",
issn = "1385-4046",
publisher = "Swets & Zeitlinger",
number = "7",

}

TY - JOUR

T1 - Reliability concerns in the repeated computerized assessment of attention in children

AU - Zabel, Theodore A

AU - Von Thomsen, Christian

AU - Cole, Carolyn

AU - Martin, Rebecca

AU - Mahone, Ernest M

PY - 2009/9

Y1 - 2009/9

N2 - Assessment of attentional processes via computerized assessment is frequently used to quantify intra-individual cognitive improvement or decline in response to treatment. However, assessment of intra-individual change is highly dependent on sufficient test reliability. We examined the test-retest reliability of selected variables from one popular computerized continuous performance test (CPT)i.e., the Conners' CPT - Second Edition (CPT-II). Participants were 39 healthy children (20 girls) ages 6-18 without intellectual impairment (mean PPVT-III SS = 102.6), LD, or psychiatric disorders (DICA-IV). Test-retest reliability over the 3-8 month interval (mean = 6 months) was acceptable (Intraclass Correlations [ICC] =.82 to.92) on comparison measures (Beery Test of Visual Perception, WISC-IV Block Design, PPVT-III). In contrast, test-retest reliability was only modest for CPT-II raw scores (ICCs ranging from.62 to.82) and T-scores (ICCs ranging from.33 to.65) for variables of interest (Omissions, Commissions, Variability, Hit Reaction Time, and Attentiveness). Using test-retest reliability information published in the CPT-II manual, 90% confidence intervals based on reliable change index (RCI) methodology were constructed to examine the significance of test-retest difference/change scores. Of the participants in this sample of typically developing youth, 30% generated intra-individual changes in T-scores on the Omissions and Attentiveness variables that exceeded the 90% confidence intervals and qualified as statistically rare changes in score. These results suggest a considerable degree of normal variability in CPT-II test scores over extended test-retest intervals, and suggest a need for caution when interpreting test score changes in neurologically unstable clinical populations.

AB - Assessment of attentional processes via computerized assessment is frequently used to quantify intra-individual cognitive improvement or decline in response to treatment. However, assessment of intra-individual change is highly dependent on sufficient test reliability. We examined the test-retest reliability of selected variables from one popular computerized continuous performance test (CPT)i.e., the Conners' CPT - Second Edition (CPT-II). Participants were 39 healthy children (20 girls) ages 6-18 without intellectual impairment (mean PPVT-III SS = 102.6), LD, or psychiatric disorders (DICA-IV). Test-retest reliability over the 3-8 month interval (mean = 6 months) was acceptable (Intraclass Correlations [ICC] =.82 to.92) on comparison measures (Beery Test of Visual Perception, WISC-IV Block Design, PPVT-III). In contrast, test-retest reliability was only modest for CPT-II raw scores (ICCs ranging from.62 to.82) and T-scores (ICCs ranging from.33 to.65) for variables of interest (Omissions, Commissions, Variability, Hit Reaction Time, and Attentiveness). Using test-retest reliability information published in the CPT-II manual, 90% confidence intervals based on reliable change index (RCI) methodology were constructed to examine the significance of test-retest difference/change scores. Of the participants in this sample of typically developing youth, 30% generated intra-individual changes in T-scores on the Omissions and Attentiveness variables that exceeded the 90% confidence intervals and qualified as statistically rare changes in score. These results suggest a considerable degree of normal variability in CPT-II test scores over extended test-retest intervals, and suggest a need for caution when interpreting test score changes in neurologically unstable clinical populations.

KW - Attention

KW - Conners' Continuous Performance Test

KW - Neuropsychology

KW - Normal development

KW - Reliable Change Index

KW - Serial assessment

UR - http://www.scopus.com/inward/record.url?scp=70349449658&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=70349449658&partnerID=8YFLogxK

U2 - 10.1080/13854040902855358

DO - 10.1080/13854040902855358

M3 - Article

VL - 23

SP - 1213

EP - 1231

JO - Clinical Neuropsychologist

JF - Clinical Neuropsychologist

SN - 1385-4046

IS - 7

ER -