Assessment of the Validity and Reliability of Three Systems of Medical Record Screening for Quality of Care Assessment

Luiz A B Camacho, Haya R. Rubin

Research output: Contribution to journalArticle

Abstract

OBJECTIVES. The authors compared the validity and reliability of three methods of medical record screening for detecting problems with quality of care of patients undergoing cardiovascular procedures: (1) the Uniform Clinical Data Set System (UCDSS), a software-based system developed by the Health Care Financing Administration (HCFA), (2) the HCFA generic screens, and (3) the Harvard Medical Practice Study (HMPS) screening criteria. METHODS. A random sample of 451 medical records of inpatients who underwent cardiovascular procedures were screened by trained nurses and were also examined by physicians masked to screening results. Physicians' judgment provided a reference standard for validity assessment of screening applied by nurses. RESULTS. Agreement between pairs of physician reviews about the quality of care was fair or slight (Kappa <0.40). Agreement between nurses about presence of any screening flags was excellent for UCDSS (0.93), slight for HCFA (0.11), and fair for HMPS (0.41). The UCDSS was more sensitive to quality problems agreed on by two physicians than HCFA or HMPS (85%, 74%, and 69%, respectively), but was also less specific (48%, 57%, and 56%, respectively). The UCDSS was much more labor-intensive with an average of 6.7 hours of nurse review per quality problem disclosed versus 2 hours required by HCFA and HMPS. CONCLUSIONS. The UCDSS computerized screens brought about substantial improvements in reliability of medical record screening, but needed to reduce its false-positive rate and to increase efficiency before it is used. From the perspective of continuous quality improvement, the UCDSS computerized algorithms provided a starting point to developing disease-specific patterns of practice to guide interventions.

Original languageEnglish (US)
Pages (from-to)748-751
Number of pages4
JournalMedical Care
Volume36
Issue number5
StatePublished - May 1998

Fingerprint

Quality of Health Care
Centers for Medicare and Medicaid Services (U.S.)
Information Systems
Reproducibility of Results
Medical Records
medical practice
health care
Nurses
nurse
Physicians
physician
Quality Improvement
Datasets
Inpatients
random sample
Software
labor
Disease
efficiency

Keywords

  • Medical record screening
  • Quality of care
  • Reliability
  • Validity

ASJC Scopus subject areas

  • Public Health, Environmental and Occupational Health
  • Nursing(all)
  • Health(social science)
  • Health Professions(all)

Cite this

Assessment of the Validity and Reliability of Three Systems of Medical Record Screening for Quality of Care Assessment. / Camacho, Luiz A B; Rubin, Haya R.

In: Medical Care, Vol. 36, No. 5, 05.1998, p. 748-751.

Research output: Contribution to journalArticle

@article{7bd7e9f41ae34a7b817ddd9af877095e,
title = "Assessment of the Validity and Reliability of Three Systems of Medical Record Screening for Quality of Care Assessment",
abstract = "OBJECTIVES. The authors compared the validity and reliability of three methods of medical record screening for detecting problems with quality of care of patients undergoing cardiovascular procedures: (1) the Uniform Clinical Data Set System (UCDSS), a software-based system developed by the Health Care Financing Administration (HCFA), (2) the HCFA generic screens, and (3) the Harvard Medical Practice Study (HMPS) screening criteria. METHODS. A random sample of 451 medical records of inpatients who underwent cardiovascular procedures were screened by trained nurses and were also examined by physicians masked to screening results. Physicians' judgment provided a reference standard for validity assessment of screening applied by nurses. RESULTS. Agreement between pairs of physician reviews about the quality of care was fair or slight (Kappa <0.40). Agreement between nurses about presence of any screening flags was excellent for UCDSS (0.93), slight for HCFA (0.11), and fair for HMPS (0.41). The UCDSS was more sensitive to quality problems agreed on by two physicians than HCFA or HMPS (85{\%}, 74{\%}, and 69{\%}, respectively), but was also less specific (48{\%}, 57{\%}, and 56{\%}, respectively). The UCDSS was much more labor-intensive with an average of 6.7 hours of nurse review per quality problem disclosed versus 2 hours required by HCFA and HMPS. CONCLUSIONS. The UCDSS computerized screens brought about substantial improvements in reliability of medical record screening, but needed to reduce its false-positive rate and to increase efficiency before it is used. From the perspective of continuous quality improvement, the UCDSS computerized algorithms provided a starting point to developing disease-specific patterns of practice to guide interventions.",
keywords = "Medical record screening, Quality of care, Reliability, Validity",
author = "Camacho, {Luiz A B} and Rubin, {Haya R.}",
year = "1998",
month = "5",
language = "English (US)",
volume = "36",
pages = "748--751",
journal = "Medical Care",
issn = "0025-7079",
publisher = "Lippincott Williams and Wilkins",
number = "5",

}

TY - JOUR

T1 - Assessment of the Validity and Reliability of Three Systems of Medical Record Screening for Quality of Care Assessment

AU - Camacho, Luiz A B

AU - Rubin, Haya R.

PY - 1998/5

Y1 - 1998/5

N2 - OBJECTIVES. The authors compared the validity and reliability of three methods of medical record screening for detecting problems with quality of care of patients undergoing cardiovascular procedures: (1) the Uniform Clinical Data Set System (UCDSS), a software-based system developed by the Health Care Financing Administration (HCFA), (2) the HCFA generic screens, and (3) the Harvard Medical Practice Study (HMPS) screening criteria. METHODS. A random sample of 451 medical records of inpatients who underwent cardiovascular procedures were screened by trained nurses and were also examined by physicians masked to screening results. Physicians' judgment provided a reference standard for validity assessment of screening applied by nurses. RESULTS. Agreement between pairs of physician reviews about the quality of care was fair or slight (Kappa <0.40). Agreement between nurses about presence of any screening flags was excellent for UCDSS (0.93), slight for HCFA (0.11), and fair for HMPS (0.41). The UCDSS was more sensitive to quality problems agreed on by two physicians than HCFA or HMPS (85%, 74%, and 69%, respectively), but was also less specific (48%, 57%, and 56%, respectively). The UCDSS was much more labor-intensive with an average of 6.7 hours of nurse review per quality problem disclosed versus 2 hours required by HCFA and HMPS. CONCLUSIONS. The UCDSS computerized screens brought about substantial improvements in reliability of medical record screening, but needed to reduce its false-positive rate and to increase efficiency before it is used. From the perspective of continuous quality improvement, the UCDSS computerized algorithms provided a starting point to developing disease-specific patterns of practice to guide interventions.

AB - OBJECTIVES. The authors compared the validity and reliability of three methods of medical record screening for detecting problems with quality of care of patients undergoing cardiovascular procedures: (1) the Uniform Clinical Data Set System (UCDSS), a software-based system developed by the Health Care Financing Administration (HCFA), (2) the HCFA generic screens, and (3) the Harvard Medical Practice Study (HMPS) screening criteria. METHODS. A random sample of 451 medical records of inpatients who underwent cardiovascular procedures were screened by trained nurses and were also examined by physicians masked to screening results. Physicians' judgment provided a reference standard for validity assessment of screening applied by nurses. RESULTS. Agreement between pairs of physician reviews about the quality of care was fair or slight (Kappa <0.40). Agreement between nurses about presence of any screening flags was excellent for UCDSS (0.93), slight for HCFA (0.11), and fair for HMPS (0.41). The UCDSS was more sensitive to quality problems agreed on by two physicians than HCFA or HMPS (85%, 74%, and 69%, respectively), but was also less specific (48%, 57%, and 56%, respectively). The UCDSS was much more labor-intensive with an average of 6.7 hours of nurse review per quality problem disclosed versus 2 hours required by HCFA and HMPS. CONCLUSIONS. The UCDSS computerized screens brought about substantial improvements in reliability of medical record screening, but needed to reduce its false-positive rate and to increase efficiency before it is used. From the perspective of continuous quality improvement, the UCDSS computerized algorithms provided a starting point to developing disease-specific patterns of practice to guide interventions.

KW - Medical record screening

KW - Quality of care

KW - Reliability

KW - Validity

UR - http://www.scopus.com/inward/record.url?scp=0032066147&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0032066147&partnerID=8YFLogxK

M3 - Article

C2 - 9596065

AN - SCOPUS:0032066147

VL - 36

SP - 748

EP - 751

JO - Medical Care

JF - Medical Care

SN - 0025-7079

IS - 5

ER -