Percentile-based empirical distribution function estimates for performance evaluation of healthcare providers

Susan M. Paddock, Thomas Louis

Research output: Contribution to journalArticle

Abstract

Hierarchical models are widely used to characterize the performance of individual healthcare providers. However, little attention has been devoted to systemwide performance evaluations, the goals of which include identifying extreme (e.g. the top 10%) provider performance and developing statistical benchmarks to define high quality care. Obtaining optimal estimates of these quantities requires estimating the empirical distribution function (EDF) of provider-specific parameters that generate the data set under consideration. However, the difficulty of obtaining uncertainty bounds for a squared error loss minimizing EDF estimate has hindered its use in systemwide performance evaluations. We therefore develop and study a percentile-based EDF estimate for univariate provider-specific parameters. We compute order statistics of samples drawn from the posterior distribution of provider-specific parameters to obtain relevant assessments of uncertainty of an EDF estimate and its features, such as thresholds and percentiles. We apply our method to data from the Medicare end stage renal disease programme, which is a health insurance programme for people with irreversible kidney failure. We highlight the risk of misclassifying providers as exceptionally good or poor performers when uncertainty in statistical benchmark estimates is ignored. Given the high stakes of performance evaluations, statistical benchmarks should be accompanied by precision estimates.

Original languageEnglish (US)
Pages (from-to)575-589
Number of pages15
JournalJournal of the Royal Statistical Society. Series C: Applied Statistics
Volume60
Issue number4
DOIs
StatePublished - Aug 2011

Fingerprint

Empirical Distribution Function
Percentile
Healthcare
Performance Evaluation
Estimate
Benchmark
Uncertainty
Squared Error Loss
Kidney
Hierarchical Model
Posterior distribution
Order Statistics
Insurance
Univariate
Distribution function
Performance evaluation
Empirical distribution
Health
Extremes

Keywords

  • Bayesian methods
  • Empirical distribution function
  • Ensemble
  • Hierarchical model
  • Statistical benchmark

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Cite this

@article{671f9f5e06cd4a88a3dfdb44cc4d5e74,
title = "Percentile-based empirical distribution function estimates for performance evaluation of healthcare providers",
abstract = "Hierarchical models are widely used to characterize the performance of individual healthcare providers. However, little attention has been devoted to systemwide performance evaluations, the goals of which include identifying extreme (e.g. the top 10{\%}) provider performance and developing statistical benchmarks to define high quality care. Obtaining optimal estimates of these quantities requires estimating the empirical distribution function (EDF) of provider-specific parameters that generate the data set under consideration. However, the difficulty of obtaining uncertainty bounds for a squared error loss minimizing EDF estimate has hindered its use in systemwide performance evaluations. We therefore develop and study a percentile-based EDF estimate for univariate provider-specific parameters. We compute order statistics of samples drawn from the posterior distribution of provider-specific parameters to obtain relevant assessments of uncertainty of an EDF estimate and its features, such as thresholds and percentiles. We apply our method to data from the Medicare end stage renal disease programme, which is a health insurance programme for people with irreversible kidney failure. We highlight the risk of misclassifying providers as exceptionally good or poor performers when uncertainty in statistical benchmark estimates is ignored. Given the high stakes of performance evaluations, statistical benchmarks should be accompanied by precision estimates.",
keywords = "Bayesian methods, Empirical distribution function, Ensemble, Hierarchical model, Statistical benchmark",
author = "Paddock, {Susan M.} and Thomas Louis",
year = "2011",
month = "8",
doi = "10.1111/j.1467-9876.2010.00760.x",
language = "English (US)",
volume = "60",
pages = "575--589",
journal = "Journal of the Royal Statistical Society. Series C: Applied Statistics",
issn = "0035-9254",
publisher = "Wiley-Blackwell",
number = "4",

}

TY - JOUR

T1 - Percentile-based empirical distribution function estimates for performance evaluation of healthcare providers

AU - Paddock, Susan M.

AU - Louis, Thomas

PY - 2011/8

Y1 - 2011/8

N2 - Hierarchical models are widely used to characterize the performance of individual healthcare providers. However, little attention has been devoted to systemwide performance evaluations, the goals of which include identifying extreme (e.g. the top 10%) provider performance and developing statistical benchmarks to define high quality care. Obtaining optimal estimates of these quantities requires estimating the empirical distribution function (EDF) of provider-specific parameters that generate the data set under consideration. However, the difficulty of obtaining uncertainty bounds for a squared error loss minimizing EDF estimate has hindered its use in systemwide performance evaluations. We therefore develop and study a percentile-based EDF estimate for univariate provider-specific parameters. We compute order statistics of samples drawn from the posterior distribution of provider-specific parameters to obtain relevant assessments of uncertainty of an EDF estimate and its features, such as thresholds and percentiles. We apply our method to data from the Medicare end stage renal disease programme, which is a health insurance programme for people with irreversible kidney failure. We highlight the risk of misclassifying providers as exceptionally good or poor performers when uncertainty in statistical benchmark estimates is ignored. Given the high stakes of performance evaluations, statistical benchmarks should be accompanied by precision estimates.

AB - Hierarchical models are widely used to characterize the performance of individual healthcare providers. However, little attention has been devoted to systemwide performance evaluations, the goals of which include identifying extreme (e.g. the top 10%) provider performance and developing statistical benchmarks to define high quality care. Obtaining optimal estimates of these quantities requires estimating the empirical distribution function (EDF) of provider-specific parameters that generate the data set under consideration. However, the difficulty of obtaining uncertainty bounds for a squared error loss minimizing EDF estimate has hindered its use in systemwide performance evaluations. We therefore develop and study a percentile-based EDF estimate for univariate provider-specific parameters. We compute order statistics of samples drawn from the posterior distribution of provider-specific parameters to obtain relevant assessments of uncertainty of an EDF estimate and its features, such as thresholds and percentiles. We apply our method to data from the Medicare end stage renal disease programme, which is a health insurance programme for people with irreversible kidney failure. We highlight the risk of misclassifying providers as exceptionally good or poor performers when uncertainty in statistical benchmark estimates is ignored. Given the high stakes of performance evaluations, statistical benchmarks should be accompanied by precision estimates.

KW - Bayesian methods

KW - Empirical distribution function

KW - Ensemble

KW - Hierarchical model

KW - Statistical benchmark

UR - http://www.scopus.com/inward/record.url?scp=79960844412&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=79960844412&partnerID=8YFLogxK

U2 - 10.1111/j.1467-9876.2010.00760.x

DO - 10.1111/j.1467-9876.2010.00760.x

M3 - Article

C2 - 21918583

AN - SCOPUS:79960844412

VL - 60

SP - 575

EP - 589

JO - Journal of the Royal Statistical Society. Series C: Applied Statistics

JF - Journal of the Royal Statistical Society. Series C: Applied Statistics

SN - 0035-9254

IS - 4

ER -