Peer-reviewed validation of a comprehensive framework for disaster evaluation typologies

Diana F. Wong, Caroline Spencer, Leanne Boyd, Frederick M. Burkle, Frank Archer

Research output: Contribution to journalArticle

Abstract

Introduction: The Comprehensive Framework for Disaster Evaluation Typologies, developed in 2017 (CFDET 2017), aims to unify and facilitate agreement regarding the identification, structure, and relationships between various evaluation typologies found in the disaster setting. A peer-reviewed validation process sought input from international experts in the fields of disaster medicine, disaster/emergency management, humanitarian/development, and evaluation. This paper discusses the validation process, its results, and outcomes.Research Problem: Previous frameworks, identified in the literature, lack validation and consistent terminology. To gain credibility and utility, this unique framework needed to be validated by international experts in the disaster setting.Methods: A mixed methods approach was designed to validate the framework. An initial iterative process informed an online survey which used a combination of a five-point Likert scale and open-ended questions. Pre-determined consensus thresholds, informed by a targeted literature review, provided the validation criteria.Results: A sample of 33 experts from 11 countries responded to the validation process. Quantitative measures largely supported the elements and relationships of the framework, and strongly supported its value and usefulness for supporting, promoting, and undertaking evaluations, as well as its usefulness for teaching evaluation in the disaster setting. Qualitative input suggested opportunities to strengthen and enhance the framework. There were limited responses to better understand the barriers and enablers of undertaking disaster evaluations. A potential for self-selection bias of respondents may be a limitation of this study. The attainment of high consensus thresholds, however, provides confidence in the validity of the results.Conclusion: For the first time, a framework of this nature has undergone a rigorous validation process by experts in three related disciplines at an international level. The modified framework, CFDET 2018, provides a unifying framework within which existing evaluation typologies can be structured. It gives evaluators confidence to choose an appropriate strategy for their particular evaluation in the disaster setting and facilitates consistency in reporting across the different phases of a disaster to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. Future research could create a series of toolkits to support improved disaster evaluation processes and to evaluate the utility of the framework in the real-world setting.

Original languageEnglish (US)
Pages (from-to)230-240
Number of pages11
JournalPrehospital and disaster medicine
Volume34
Issue number3
DOIs
StatePublished - Jun 1 2019

Fingerprint

Disasters
Disaster Medicine
Selection Bias
Terminology
Reproducibility of Results
Teaching
Emergencies
Research

Keywords

  • disasters
  • evaluation
  • framework
  • validation

ASJC Scopus subject areas

  • Emergency Medicine
  • Emergency

Cite this

Peer-reviewed validation of a comprehensive framework for disaster evaluation typologies. / Wong, Diana F.; Spencer, Caroline; Boyd, Leanne; Burkle, Frederick M.; Archer, Frank.

In: Prehospital and disaster medicine, Vol. 34, No. 3, 01.06.2019, p. 230-240.

Research output: Contribution to journalArticle

Wong, Diana F. ; Spencer, Caroline ; Boyd, Leanne ; Burkle, Frederick M. ; Archer, Frank. / Peer-reviewed validation of a comprehensive framework for disaster evaluation typologies. In: Prehospital and disaster medicine. 2019 ; Vol. 34, No. 3. pp. 230-240.
@article{18ee092d7f0743ee9b1c31add19dca6e,
title = "Peer-reviewed validation of a comprehensive framework for disaster evaluation typologies",
abstract = "Introduction: The Comprehensive Framework for Disaster Evaluation Typologies, developed in 2017 (CFDET 2017), aims to unify and facilitate agreement regarding the identification, structure, and relationships between various evaluation typologies found in the disaster setting. A peer-reviewed validation process sought input from international experts in the fields of disaster medicine, disaster/emergency management, humanitarian/development, and evaluation. This paper discusses the validation process, its results, and outcomes.Research Problem: Previous frameworks, identified in the literature, lack validation and consistent terminology. To gain credibility and utility, this unique framework needed to be validated by international experts in the disaster setting.Methods: A mixed methods approach was designed to validate the framework. An initial iterative process informed an online survey which used a combination of a five-point Likert scale and open-ended questions. Pre-determined consensus thresholds, informed by a targeted literature review, provided the validation criteria.Results: A sample of 33 experts from 11 countries responded to the validation process. Quantitative measures largely supported the elements and relationships of the framework, and strongly supported its value and usefulness for supporting, promoting, and undertaking evaluations, as well as its usefulness for teaching evaluation in the disaster setting. Qualitative input suggested opportunities to strengthen and enhance the framework. There were limited responses to better understand the barriers and enablers of undertaking disaster evaluations. A potential for self-selection bias of respondents may be a limitation of this study. The attainment of high consensus thresholds, however, provides confidence in the validity of the results.Conclusion: For the first time, a framework of this nature has undergone a rigorous validation process by experts in three related disciplines at an international level. The modified framework, CFDET 2018, provides a unifying framework within which existing evaluation typologies can be structured. It gives evaluators confidence to choose an appropriate strategy for their particular evaluation in the disaster setting and facilitates consistency in reporting across the different phases of a disaster to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. Future research could create a series of toolkits to support improved disaster evaluation processes and to evaluate the utility of the framework in the real-world setting.",
keywords = "disasters, evaluation, framework, validation",
author = "Wong, {Diana F.} and Caroline Spencer and Leanne Boyd and Burkle, {Frederick M.} and Frank Archer",
year = "2019",
month = "6",
day = "1",
doi = "10.1017/S1049023X19004400",
language = "English (US)",
volume = "34",
pages = "230--240",
journal = "Prehospital and Disaster Medicine",
issn = "1049-023X",
publisher = "World Association for Disaster and Emergency Medicine",
number = "3",

}

TY - JOUR

T1 - Peer-reviewed validation of a comprehensive framework for disaster evaluation typologies

AU - Wong, Diana F.

AU - Spencer, Caroline

AU - Boyd, Leanne

AU - Burkle, Frederick M.

AU - Archer, Frank

PY - 2019/6/1

Y1 - 2019/6/1

N2 - Introduction: The Comprehensive Framework for Disaster Evaluation Typologies, developed in 2017 (CFDET 2017), aims to unify and facilitate agreement regarding the identification, structure, and relationships between various evaluation typologies found in the disaster setting. A peer-reviewed validation process sought input from international experts in the fields of disaster medicine, disaster/emergency management, humanitarian/development, and evaluation. This paper discusses the validation process, its results, and outcomes.Research Problem: Previous frameworks, identified in the literature, lack validation and consistent terminology. To gain credibility and utility, this unique framework needed to be validated by international experts in the disaster setting.Methods: A mixed methods approach was designed to validate the framework. An initial iterative process informed an online survey which used a combination of a five-point Likert scale and open-ended questions. Pre-determined consensus thresholds, informed by a targeted literature review, provided the validation criteria.Results: A sample of 33 experts from 11 countries responded to the validation process. Quantitative measures largely supported the elements and relationships of the framework, and strongly supported its value and usefulness for supporting, promoting, and undertaking evaluations, as well as its usefulness for teaching evaluation in the disaster setting. Qualitative input suggested opportunities to strengthen and enhance the framework. There were limited responses to better understand the barriers and enablers of undertaking disaster evaluations. A potential for self-selection bias of respondents may be a limitation of this study. The attainment of high consensus thresholds, however, provides confidence in the validity of the results.Conclusion: For the first time, a framework of this nature has undergone a rigorous validation process by experts in three related disciplines at an international level. The modified framework, CFDET 2018, provides a unifying framework within which existing evaluation typologies can be structured. It gives evaluators confidence to choose an appropriate strategy for their particular evaluation in the disaster setting and facilitates consistency in reporting across the different phases of a disaster to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. Future research could create a series of toolkits to support improved disaster evaluation processes and to evaluate the utility of the framework in the real-world setting.

AB - Introduction: The Comprehensive Framework for Disaster Evaluation Typologies, developed in 2017 (CFDET 2017), aims to unify and facilitate agreement regarding the identification, structure, and relationships between various evaluation typologies found in the disaster setting. A peer-reviewed validation process sought input from international experts in the fields of disaster medicine, disaster/emergency management, humanitarian/development, and evaluation. This paper discusses the validation process, its results, and outcomes.Research Problem: Previous frameworks, identified in the literature, lack validation and consistent terminology. To gain credibility and utility, this unique framework needed to be validated by international experts in the disaster setting.Methods: A mixed methods approach was designed to validate the framework. An initial iterative process informed an online survey which used a combination of a five-point Likert scale and open-ended questions. Pre-determined consensus thresholds, informed by a targeted literature review, provided the validation criteria.Results: A sample of 33 experts from 11 countries responded to the validation process. Quantitative measures largely supported the elements and relationships of the framework, and strongly supported its value and usefulness for supporting, promoting, and undertaking evaluations, as well as its usefulness for teaching evaluation in the disaster setting. Qualitative input suggested opportunities to strengthen and enhance the framework. There were limited responses to better understand the barriers and enablers of undertaking disaster evaluations. A potential for self-selection bias of respondents may be a limitation of this study. The attainment of high consensus thresholds, however, provides confidence in the validity of the results.Conclusion: For the first time, a framework of this nature has undergone a rigorous validation process by experts in three related disciplines at an international level. The modified framework, CFDET 2018, provides a unifying framework within which existing evaluation typologies can be structured. It gives evaluators confidence to choose an appropriate strategy for their particular evaluation in the disaster setting and facilitates consistency in reporting across the different phases of a disaster to better understand the process, outcomes, and impacts of the efficacy and efficiency of interventions. Future research could create a series of toolkits to support improved disaster evaluation processes and to evaluate the utility of the framework in the real-world setting.

KW - disasters

KW - evaluation

KW - framework

KW - validation

UR - http://www.scopus.com/inward/record.url?scp=85067414487&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067414487&partnerID=8YFLogxK

U2 - 10.1017/S1049023X19004400

DO - 10.1017/S1049023X19004400

M3 - Article

VL - 34

SP - 230

EP - 240

JO - Prehospital and Disaster Medicine

JF - Prehospital and Disaster Medicine

SN - 1049-023X

IS - 3

ER -