Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol

Jason E. Chapman, Michael R. Mccart, Elizabeth J Letourneau, Ashli J. Sheidow

Research output: Contribution to journalArticle

Abstract

Objective: This study evaluated the accuracy of youth, caregiver, therapist, and trained raters relative to treatment experts on ratings of therapist adherence to a substance abuse treatment protocol for adolescents. Method: Adherence ratings were provided by youth and caregivers in an ongoing trial evaluating a Contingency Management (CM) intervention for youth in juvenile drug court. These ratings were compared to those provided by therapists and trained raters, and each rater type was compared to ratings provided by CM treatment experts. Data were analyzed using item-response-theory-based Many-Facet Rasch Models. Results: Relative to treatment experts, youth and caregivers were significantly more likely to endorse the occurrence of CM components. In contrast, therapists and trained raters were much more consistent with treatment experts. In terms of practical significance, youth and caregivers each had a 97% estimated probability of indicating that a typical treatment component had occurred. By comparison, the probability was 31%, 19%, and 26% for therapists, trained raters, and treatment experts, respectively. Conclusions: Youth and caregivers were highly inaccurate relative to treatment experts, whereas, therapists and trained raters were generally consistent with treatment experts. The implications of these findings for therapist adherence measurement are considered.

Original languageEnglish (US)
Pages (from-to)674-680
Number of pages7
JournalJournal of Consulting and Clinical Psychology
Volume81
Issue number4
DOIs
StatePublished - Aug 2013

Fingerprint

Clinical Protocols
Caregivers
Substance-Related Disorders
Therapeutics
Raters
Substance Abuse
Adherence
Rating
Pharmaceutical Preparations

Keywords

  • contingency management
  • rater method
  • substance use
  • therapist adherence

ASJC Scopus subject areas

  • Psychiatry and Mental health
  • Clinical Psychology
  • Arts and Humanities (miscellaneous)
  • Medicine(all)

Cite this

Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol. / Chapman, Jason E.; Mccart, Michael R.; Letourneau, Elizabeth J; Sheidow, Ashli J.

In: Journal of Consulting and Clinical Psychology, Vol. 81, No. 4, 08.2013, p. 674-680.

Research output: Contribution to journalArticle

@article{a113f5d4348640eda575b08936193cfe,
title = "Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol",
abstract = "Objective: This study evaluated the accuracy of youth, caregiver, therapist, and trained raters relative to treatment experts on ratings of therapist adherence to a substance abuse treatment protocol for adolescents. Method: Adherence ratings were provided by youth and caregivers in an ongoing trial evaluating a Contingency Management (CM) intervention for youth in juvenile drug court. These ratings were compared to those provided by therapists and trained raters, and each rater type was compared to ratings provided by CM treatment experts. Data were analyzed using item-response-theory-based Many-Facet Rasch Models. Results: Relative to treatment experts, youth and caregivers were significantly more likely to endorse the occurrence of CM components. In contrast, therapists and trained raters were much more consistent with treatment experts. In terms of practical significance, youth and caregivers each had a 97{\%} estimated probability of indicating that a typical treatment component had occurred. By comparison, the probability was 31{\%}, 19{\%}, and 26{\%} for therapists, trained raters, and treatment experts, respectively. Conclusions: Youth and caregivers were highly inaccurate relative to treatment experts, whereas, therapists and trained raters were generally consistent with treatment experts. The implications of these findings for therapist adherence measurement are considered.",
keywords = "contingency management, rater method, substance use, therapist adherence",
author = "Chapman, {Jason E.} and Mccart, {Michael R.} and Letourneau, {Elizabeth J} and Sheidow, {Ashli J.}",
year = "2013",
month = "8",
doi = "10.1037/a0033021",
language = "English (US)",
volume = "81",
pages = "674--680",
journal = "Journal of Consulting and Clinical Psychology",
issn = "0022-006X",
publisher = "American Psychological Association Inc.",
number = "4",

}

TY - JOUR

T1 - Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol

AU - Chapman, Jason E.

AU - Mccart, Michael R.

AU - Letourneau, Elizabeth J

AU - Sheidow, Ashli J.

PY - 2013/8

Y1 - 2013/8

N2 - Objective: This study evaluated the accuracy of youth, caregiver, therapist, and trained raters relative to treatment experts on ratings of therapist adherence to a substance abuse treatment protocol for adolescents. Method: Adherence ratings were provided by youth and caregivers in an ongoing trial evaluating a Contingency Management (CM) intervention for youth in juvenile drug court. These ratings were compared to those provided by therapists and trained raters, and each rater type was compared to ratings provided by CM treatment experts. Data were analyzed using item-response-theory-based Many-Facet Rasch Models. Results: Relative to treatment experts, youth and caregivers were significantly more likely to endorse the occurrence of CM components. In contrast, therapists and trained raters were much more consistent with treatment experts. In terms of practical significance, youth and caregivers each had a 97% estimated probability of indicating that a typical treatment component had occurred. By comparison, the probability was 31%, 19%, and 26% for therapists, trained raters, and treatment experts, respectively. Conclusions: Youth and caregivers were highly inaccurate relative to treatment experts, whereas, therapists and trained raters were generally consistent with treatment experts. The implications of these findings for therapist adherence measurement are considered.

AB - Objective: This study evaluated the accuracy of youth, caregiver, therapist, and trained raters relative to treatment experts on ratings of therapist adherence to a substance abuse treatment protocol for adolescents. Method: Adherence ratings were provided by youth and caregivers in an ongoing trial evaluating a Contingency Management (CM) intervention for youth in juvenile drug court. These ratings were compared to those provided by therapists and trained raters, and each rater type was compared to ratings provided by CM treatment experts. Data were analyzed using item-response-theory-based Many-Facet Rasch Models. Results: Relative to treatment experts, youth and caregivers were significantly more likely to endorse the occurrence of CM components. In contrast, therapists and trained raters were much more consistent with treatment experts. In terms of practical significance, youth and caregivers each had a 97% estimated probability of indicating that a typical treatment component had occurred. By comparison, the probability was 31%, 19%, and 26% for therapists, trained raters, and treatment experts, respectively. Conclusions: Youth and caregivers were highly inaccurate relative to treatment experts, whereas, therapists and trained raters were generally consistent with treatment experts. The implications of these findings for therapist adherence measurement are considered.

KW - contingency management

KW - rater method

KW - substance use

KW - therapist adherence

UR - http://www.scopus.com/inward/record.url?scp=84881067112&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84881067112&partnerID=8YFLogxK

U2 - 10.1037/a0033021

DO - 10.1037/a0033021

M3 - Article

C2 - 23668668

AN - SCOPUS:84881067112

VL - 81

SP - 674

EP - 680

JO - Journal of Consulting and Clinical Psychology

JF - Journal of Consulting and Clinical Psychology

SN - 0022-006X

IS - 4

ER -