Diagnostic Measures for Model Criticism

Cinzia Carota, Giovanni Parmigiani, Nicholas G. Polson

Research output: Contribution to journalArticle

Abstract

We discuss the problem of model criticism, with emphasis on developing summary diagnostic measures. We approach model criticism by identifying possible troublesome features of the currently entertained model, embedding the model in an elaborated model, and measuring the value of elaborating. This requires three elements: a model elaboration, a prior distribution, and a utility function. Each triplet generates a different diagnostic measure. We focus primarily on the measure given by a Kullback-Leibler divergence between the marginal prior and posterior distributions on the elaboration parameter. We also develop a linearized version of this diagnostic and use it to show that our procedure is related to other tools commonly used for model diagnostics, such as Bayes factors and the score function. One attraction of this approach is that it allows model criticism to be performed jointly with parameter inference and prediction. Also, this diagnostic approach aims at maintaining an exploratory nature to the criticism process, while affording feasibility of implementation. In this article we present the general outlook and discuss general families of elaborations for use in practice; the exponential connection elaboration plays a key role. We then describe model elaborations for use in diagnosing: departures from normality, goodness of fit in generalized linear models, and variable selection in regression and outlier detection. We illustrate our approach with two applications.

Original languageEnglish (US)
Pages (from-to)753-762
Number of pages10
JournalJournal of the American Statistical Association
Volume91
Issue number434
StatePublished - Jun 1996
Externally publishedYes

Fingerprint

Diagnostics
Prior distribution
Model
Model Diagnostics
Bayes Factor
Kullback-Leibler Divergence
Score Function
Criticism
Outlier Detection
Generalized Linear Model
Variable Selection
Marginal Distribution
Goodness of fit
Posterior distribution
Utility Function
Model Selection
Normality
Regression
Elaboration
Prediction

Keywords

  • Bayes factor
  • Extra-Poisson variability
  • Hierarchical models
  • Information
  • Model influence
  • Outliers
  • Prior influence
  • Score function

ASJC Scopus subject areas

  • Mathematics(all)
  • Statistics and Probability

Cite this

Carota, C., Parmigiani, G., & Polson, N. G. (1996). Diagnostic Measures for Model Criticism. Journal of the American Statistical Association, 91(434), 753-762.

Diagnostic Measures for Model Criticism. / Carota, Cinzia; Parmigiani, Giovanni; Polson, Nicholas G.

In: Journal of the American Statistical Association, Vol. 91, No. 434, 06.1996, p. 753-762.

Research output: Contribution to journalArticle

Carota, C, Parmigiani, G & Polson, NG 1996, 'Diagnostic Measures for Model Criticism', Journal of the American Statistical Association, vol. 91, no. 434, pp. 753-762.
Carota C, Parmigiani G, Polson NG. Diagnostic Measures for Model Criticism. Journal of the American Statistical Association. 1996 Jun;91(434):753-762.
Carota, Cinzia ; Parmigiani, Giovanni ; Polson, Nicholas G. / Diagnostic Measures for Model Criticism. In: Journal of the American Statistical Association. 1996 ; Vol. 91, No. 434. pp. 753-762.
@article{543bd742535843e3a8222f2f17d52b52,
title = "Diagnostic Measures for Model Criticism",
abstract = "We discuss the problem of model criticism, with emphasis on developing summary diagnostic measures. We approach model criticism by identifying possible troublesome features of the currently entertained model, embedding the model in an elaborated model, and measuring the value of elaborating. This requires three elements: a model elaboration, a prior distribution, and a utility function. Each triplet generates a different diagnostic measure. We focus primarily on the measure given by a Kullback-Leibler divergence between the marginal prior and posterior distributions on the elaboration parameter. We also develop a linearized version of this diagnostic and use it to show that our procedure is related to other tools commonly used for model diagnostics, such as Bayes factors and the score function. One attraction of this approach is that it allows model criticism to be performed jointly with parameter inference and prediction. Also, this diagnostic approach aims at maintaining an exploratory nature to the criticism process, while affording feasibility of implementation. In this article we present the general outlook and discuss general families of elaborations for use in practice; the exponential connection elaboration plays a key role. We then describe model elaborations for use in diagnosing: departures from normality, goodness of fit in generalized linear models, and variable selection in regression and outlier detection. We illustrate our approach with two applications.",
keywords = "Bayes factor, Extra-Poisson variability, Hierarchical models, Information, Model influence, Outliers, Prior influence, Score function",
author = "Cinzia Carota and Giovanni Parmigiani and Polson, {Nicholas G.}",
year = "1996",
month = "6",
language = "English (US)",
volume = "91",
pages = "753--762",
journal = "Journal of the American Statistical Association",
issn = "0162-1459",
publisher = "Taylor and Francis Ltd.",
number = "434",

}

TY - JOUR

T1 - Diagnostic Measures for Model Criticism

AU - Carota, Cinzia

AU - Parmigiani, Giovanni

AU - Polson, Nicholas G.

PY - 1996/6

Y1 - 1996/6

N2 - We discuss the problem of model criticism, with emphasis on developing summary diagnostic measures. We approach model criticism by identifying possible troublesome features of the currently entertained model, embedding the model in an elaborated model, and measuring the value of elaborating. This requires three elements: a model elaboration, a prior distribution, and a utility function. Each triplet generates a different diagnostic measure. We focus primarily on the measure given by a Kullback-Leibler divergence between the marginal prior and posterior distributions on the elaboration parameter. We also develop a linearized version of this diagnostic and use it to show that our procedure is related to other tools commonly used for model diagnostics, such as Bayes factors and the score function. One attraction of this approach is that it allows model criticism to be performed jointly with parameter inference and prediction. Also, this diagnostic approach aims at maintaining an exploratory nature to the criticism process, while affording feasibility of implementation. In this article we present the general outlook and discuss general families of elaborations for use in practice; the exponential connection elaboration plays a key role. We then describe model elaborations for use in diagnosing: departures from normality, goodness of fit in generalized linear models, and variable selection in regression and outlier detection. We illustrate our approach with two applications.

AB - We discuss the problem of model criticism, with emphasis on developing summary diagnostic measures. We approach model criticism by identifying possible troublesome features of the currently entertained model, embedding the model in an elaborated model, and measuring the value of elaborating. This requires three elements: a model elaboration, a prior distribution, and a utility function. Each triplet generates a different diagnostic measure. We focus primarily on the measure given by a Kullback-Leibler divergence between the marginal prior and posterior distributions on the elaboration parameter. We also develop a linearized version of this diagnostic and use it to show that our procedure is related to other tools commonly used for model diagnostics, such as Bayes factors and the score function. One attraction of this approach is that it allows model criticism to be performed jointly with parameter inference and prediction. Also, this diagnostic approach aims at maintaining an exploratory nature to the criticism process, while affording feasibility of implementation. In this article we present the general outlook and discuss general families of elaborations for use in practice; the exponential connection elaboration plays a key role. We then describe model elaborations for use in diagnosing: departures from normality, goodness of fit in generalized linear models, and variable selection in regression and outlier detection. We illustrate our approach with two applications.

KW - Bayes factor

KW - Extra-Poisson variability

KW - Hierarchical models

KW - Information

KW - Model influence

KW - Outliers

KW - Prior influence

KW - Score function

UR - http://www.scopus.com/inward/record.url?scp=0030546466&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0030546466&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:0030546466

VL - 91

SP - 753

EP - 762

JO - Journal of the American Statistical Association

JF - Journal of the American Statistical Association

SN - 0162-1459

IS - 434

ER -