Development of an itemwise efficiency scoring method: Concurrent, convergent, discriminant, and neuroimaging-based predictive validity assessed in a large community sample

Tyler M. Moore, Steven P. Reise, David R. Roalf, Theodore D. Satterthwaite, Christos Davatzikos, Warren B. Bilker, Allison M. Port, Chad T. Jackson, Kosha Ruparel, Adam P. Savitt, Robert B. Baron, Raquel E. Gur, Ruben C. Gur

Research output: Contribution to journalArticlepeer-review

Abstract

Traditional "paper-and-pencil" testing is imprecise in measuring speed and hence limited in assessing performance efficiency, but computerized testing permits precision in measuring itemwise response time. We present a method of scoring performance efficiency (combining information from accuracy and speed) at the item level. Using a community sample of 9,498 youths age 8-21, we calculated item-level efficiency scores on 4 neurocognitive tests, and compared the concurrent, convergent, discriminant, and predictive validity of these scores with simple averaging of standardized speed and accuracy-summed scores. Concurrent validity was measured by the scores' abilities to distinguish men from women and their correlations with age; convergent and discriminant validity were measured by correlations with other scores inside and outside of their neurocognitive domains; predictive validity was measured by correlations with brain volume in regions associated with the specific neurocognitive abilities. Results provide support for the ability of itemwise efficiency scoring to detect signals as strong as those detected by standard efficiency scoring methods. We find no evidence of superior validity of the itemwise scores over traditional scores, but point out several advantages of the former. The itemwise efficiency scoring method shows promise as an alternative to standard efficiency scoring methods, with overall moderate support from tests of 4 different types of validity. This method allows the use of existing item analysis methods and provides the convenient ability to adjust the overall emphasis of accuracy versus speed in the efficiency score, thus adjusting the scoring to the real-world demands the test is aiming to fulfill.

Original languageEnglish (US)
Pages (from-to)1529-1542
Number of pages14
JournalPsychological Assessment
Volume28
Issue number12
DOIs
StatePublished - Dec 1 2016

Keywords

  • Computerized neurocognitive battery
  • Neurocognitive efficiency
  • Philadelphia Neurodevelopmental Cohort
  • Psychometrics
  • Validity

ASJC Scopus subject areas

  • Clinical Psychology
  • Psychiatry and Mental health

Fingerprint

Dive into the research topics of 'Development of an itemwise efficiency scoring method: Concurrent, convergent, discriminant, and neuroimaging-based predictive validity assessed in a large community sample'. Together they form a unique fingerprint.

Cite this