Watching the Doctor-Watchers

How Well Do Peer Review Organization Methods Detect Hospital Care Quality Problems?

Haya R. Rubin, William H. Rogers, Katherine L. Kahn, Lisa V. Rubenstein, Robert H. Brook

Research output: Contribution to journalArticle

Abstract

Objective. —To determine how well one state’s peer review organization (PRO) judged the quality of hospital care compared with an independent, credible judgment of quality of care. Design. —Retrospective study comparing a PRO’s review, including initial screening, physician review, and final judgments, with an independent “study judgment” based on blinded, structured, implicit reviews of hospital records. Setting. —One state’s medical and surgical Medicare hospitalizations during 1985 through 1987 audited randomly by the state’s PRO. Sample. —Stratified random sampling of records: 62 records that passed the PRO initial screening process and were not referred for PRO physician review; 50 records that failed PRO screen and then were confirmed by PRO physicians to be “quality problems.” Intervention. —None. Main Outcome Measure. —A study judgment of below standard or standard or above based on the mean of overall ratings by five internists for records in medical diagnosis related groups (DRGs) and by five internists and five surgeons for surgical DRGs. Each step in the PRO review was evaluated for how many records passing or failing that step were judged standard or above or below standard in the study (positive and negative predictive value) and how well that step classified records that the study judged below standard or standard or above (sensitivity and specificity). Results. —An estimated 18% of records reviewed by the PRO were below standard according to the study judgment, compared with 6.3% quality problems according to the PRO’s final judgment (difference, 12%; 95% confidence interval, 1 to 23). The PRO’s initial screening process failed to detect and refer for PRO physician review two of three records that the study judged below standard. In addition, only one of three of the records that PRO physicians judged to be quality problems were judged below standard by the study judgment. Therefore, the PRO’s final quality of care judgment and the study judgment agreed little more than expected by chance, especially about poor quality of care. Although the PRO correctly classified 95% of the records that the study judged standard or above, it detected only 11% of records judged below standard by the study. Conclusions. —Most of all, this PRO review process would be improved by additional preliminary screens to identify the 67% of records that the study judged below standard but that passed its initial screening. The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through improved reviewer selection and training, a structured review method, and more physician reviewers per record.

Original languageEnglish (US)
Pages (from-to)2349-2354
Number of pages6
JournalJournal of the American Medical Association
Volume267
Issue number17
DOIs
StatePublished - May 6 1992

Fingerprint

Professional Review Organizations
Quality of Health Care
Physicians
Diagnosis-Related Groups
Hospital Records
Medicare

ASJC Scopus subject areas

  • Medicine(all)

Cite this

Watching the Doctor-Watchers : How Well Do Peer Review Organization Methods Detect Hospital Care Quality Problems? / Rubin, Haya R.; Rogers, William H.; Kahn, Katherine L.; Rubenstein, Lisa V.; Brook, Robert H.

In: Journal of the American Medical Association, Vol. 267, No. 17, 06.05.1992, p. 2349-2354.

Research output: Contribution to journalArticle

Rubin, Haya R. ; Rogers, William H. ; Kahn, Katherine L. ; Rubenstein, Lisa V. ; Brook, Robert H. / Watching the Doctor-Watchers : How Well Do Peer Review Organization Methods Detect Hospital Care Quality Problems?. In: Journal of the American Medical Association. 1992 ; Vol. 267, No. 17. pp. 2349-2354.
@article{40fdc8238c034e4b8145c16800336912,
title = "Watching the Doctor-Watchers: How Well Do Peer Review Organization Methods Detect Hospital Care Quality Problems?",
abstract = "Objective. —To determine how well one state’s peer review organization (PRO) judged the quality of hospital care compared with an independent, credible judgment of quality of care. Design. —Retrospective study comparing a PRO’s review, including initial screening, physician review, and final judgments, with an independent “study judgment” based on blinded, structured, implicit reviews of hospital records. Setting. —One state’s medical and surgical Medicare hospitalizations during 1985 through 1987 audited randomly by the state’s PRO. Sample. —Stratified random sampling of records: 62 records that passed the PRO initial screening process and were not referred for PRO physician review; 50 records that failed PRO screen and then were confirmed by PRO physicians to be “quality problems.” Intervention. —None. Main Outcome Measure. —A study judgment of below standard or standard or above based on the mean of overall ratings by five internists for records in medical diagnosis related groups (DRGs) and by five internists and five surgeons for surgical DRGs. Each step in the PRO review was evaluated for how many records passing or failing that step were judged standard or above or below standard in the study (positive and negative predictive value) and how well that step classified records that the study judged below standard or standard or above (sensitivity and specificity). Results. —An estimated 18{\%} of records reviewed by the PRO were below standard according to the study judgment, compared with 6.3{\%} quality problems according to the PRO’s final judgment (difference, 12{\%}; 95{\%} confidence interval, 1 to 23). The PRO’s initial screening process failed to detect and refer for PRO physician review two of three records that the study judged below standard. In addition, only one of three of the records that PRO physicians judged to be quality problems were judged below standard by the study judgment. Therefore, the PRO’s final quality of care judgment and the study judgment agreed little more than expected by chance, especially about poor quality of care. Although the PRO correctly classified 95{\%} of the records that the study judged standard or above, it detected only 11{\%} of records judged below standard by the study. Conclusions. —Most of all, this PRO review process would be improved by additional preliminary screens to identify the 67{\%} of records that the study judged below standard but that passed its initial screening. The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through improved reviewer selection and training, a structured review method, and more physician reviewers per record.",
author = "Rubin, {Haya R.} and Rogers, {William H.} and Kahn, {Katherine L.} and Rubenstein, {Lisa V.} and Brook, {Robert H.}",
year = "1992",
month = "5",
day = "6",
doi = "10.1001/jama.1992.03480170075032",
language = "English (US)",
volume = "267",
pages = "2349--2354",
journal = "JAMA - Journal of the American Medical Association",
issn = "0098-7484",
publisher = "American Medical Association",
number = "17",

}

TY - JOUR

T1 - Watching the Doctor-Watchers

T2 - How Well Do Peer Review Organization Methods Detect Hospital Care Quality Problems?

AU - Rubin, Haya R.

AU - Rogers, William H.

AU - Kahn, Katherine L.

AU - Rubenstein, Lisa V.

AU - Brook, Robert H.

PY - 1992/5/6

Y1 - 1992/5/6

N2 - Objective. —To determine how well one state’s peer review organization (PRO) judged the quality of hospital care compared with an independent, credible judgment of quality of care. Design. —Retrospective study comparing a PRO’s review, including initial screening, physician review, and final judgments, with an independent “study judgment” based on blinded, structured, implicit reviews of hospital records. Setting. —One state’s medical and surgical Medicare hospitalizations during 1985 through 1987 audited randomly by the state’s PRO. Sample. —Stratified random sampling of records: 62 records that passed the PRO initial screening process and were not referred for PRO physician review; 50 records that failed PRO screen and then were confirmed by PRO physicians to be “quality problems.” Intervention. —None. Main Outcome Measure. —A study judgment of below standard or standard or above based on the mean of overall ratings by five internists for records in medical diagnosis related groups (DRGs) and by five internists and five surgeons for surgical DRGs. Each step in the PRO review was evaluated for how many records passing or failing that step were judged standard or above or below standard in the study (positive and negative predictive value) and how well that step classified records that the study judged below standard or standard or above (sensitivity and specificity). Results. —An estimated 18% of records reviewed by the PRO were below standard according to the study judgment, compared with 6.3% quality problems according to the PRO’s final judgment (difference, 12%; 95% confidence interval, 1 to 23). The PRO’s initial screening process failed to detect and refer for PRO physician review two of three records that the study judged below standard. In addition, only one of three of the records that PRO physicians judged to be quality problems were judged below standard by the study judgment. Therefore, the PRO’s final quality of care judgment and the study judgment agreed little more than expected by chance, especially about poor quality of care. Although the PRO correctly classified 95% of the records that the study judged standard or above, it detected only 11% of records judged below standard by the study. Conclusions. —Most of all, this PRO review process would be improved by additional preliminary screens to identify the 67% of records that the study judged below standard but that passed its initial screening. The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through improved reviewer selection and training, a structured review method, and more physician reviewers per record.

AB - Objective. —To determine how well one state’s peer review organization (PRO) judged the quality of hospital care compared with an independent, credible judgment of quality of care. Design. —Retrospective study comparing a PRO’s review, including initial screening, physician review, and final judgments, with an independent “study judgment” based on blinded, structured, implicit reviews of hospital records. Setting. —One state’s medical and surgical Medicare hospitalizations during 1985 through 1987 audited randomly by the state’s PRO. Sample. —Stratified random sampling of records: 62 records that passed the PRO initial screening process and were not referred for PRO physician review; 50 records that failed PRO screen and then were confirmed by PRO physicians to be “quality problems.” Intervention. —None. Main Outcome Measure. —A study judgment of below standard or standard or above based on the mean of overall ratings by five internists for records in medical diagnosis related groups (DRGs) and by five internists and five surgeons for surgical DRGs. Each step in the PRO review was evaluated for how many records passing or failing that step were judged standard or above or below standard in the study (positive and negative predictive value) and how well that step classified records that the study judged below standard or standard or above (sensitivity and specificity). Results. —An estimated 18% of records reviewed by the PRO were below standard according to the study judgment, compared with 6.3% quality problems according to the PRO’s final judgment (difference, 12%; 95% confidence interval, 1 to 23). The PRO’s initial screening process failed to detect and refer for PRO physician review two of three records that the study judged below standard. In addition, only one of three of the records that PRO physicians judged to be quality problems were judged below standard by the study judgment. Therefore, the PRO’s final quality of care judgment and the study judgment agreed little more than expected by chance, especially about poor quality of care. Although the PRO correctly classified 95% of the records that the study judged standard or above, it detected only 11% of records judged below standard by the study. Conclusions. —Most of all, this PRO review process would be improved by additional preliminary screens to identify the 67% of records that the study judged below standard but that passed its initial screening. The screening process also must be more accurate in order to be cost-effective, as it was only slightly better than random sampling at correctly identifying below standard care. More reproducible physician review is also needed and might be accomplished through improved reviewer selection and training, a structured review method, and more physician reviewers per record.

UR - http://www.scopus.com/inward/record.url?scp=0026549320&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0026549320&partnerID=8YFLogxK

U2 - 10.1001/jama.1992.03480170075032

DO - 10.1001/jama.1992.03480170075032

M3 - Article

VL - 267

SP - 2349

EP - 2354

JO - JAMA - Journal of the American Medical Association

JF - JAMA - Journal of the American Medical Association

SN - 0098-7484

IS - 17

ER -