Evaluation of previously embolized intracranial aneurysms: Inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists

Scott L. Zuckerman, Nikita Lakomkin, Jordan A. Magarik, Jan Vargas, Marcus Stephens, Babatunde Akinpelu, Alejandro M. Spiotta, Azam Ahmed, Adam S. Arthur, David Fiorella, Ricardo Hanel, Joshua A. Hirsch, Ferdinand Hui, Robert F. James, David F. Kallmes, Philip M. Meyers, David B. Niemann, Peter Rasmussen, Raymond D. Turner, Babu G. WelchJ. Mocco

Research output: Contribution to journalArticle

Abstract

Background The angiographic evaluation of previously coiled aneurysms can be difficult yet remains critical for determining re-treatment. Objective The main objective of this study was to determine the inter-rater reliability for both the Raymond Scale and per cent embolization among a group of neurointerventionalists evaluating previously embolized aneurysms. Methods A panel of 15 neurointerventionalists examined 92 distinct cases of immediate post-coil embolization and 1 year post-embolization angiographs. Each case was presented four times throughout the study, along with alterations in demographics in order to evaluate intra-rater reliability. All respondents were asked to provide the per cent embolization (0-100%) and Raymond Scale grade (1-3) for each aneurysm. Inter-rater reliability was evaluated by computing weighted kappa values (for the Raymond Scale) and intraclass correlation coefficients (ICC) for per cent embolization. Results 10 neurosurgeons and 5 interventional neuroradiologists evaluated 368 simulated cases. The agreement among all readers employing the Raymond Scale was fair (κ =0.35) while concordance in per cent embolization was good (ICC=0.64). Clinicians with fewer than 10 years of experience demonstrated a significantly greater level of agreement than the group with greater than 10 years (κ =0.39 and ICC=0.70 vs κ =0.28 and ICC=0.58). When the same aneurysm was presented multiple times, clinicians demonstrated excellent consistency when assessing per cent embolization (ICC=0.82), but moderate agreement when employing the Raymond classification (κ =0.58). Conclusions Identifying the per cent embolization in previously coiled aneurysms resulted in good inter- and intra-rater agreement, regardless of years of experience. The strong agreement among providers employing per cent embolization may make it a valuable tool for embolization assessment in this patient population.

Original languageEnglish (US)
Pages (from-to)464-468
Number of pages5
JournalJournal of NeuroInterventional Surgery
Volume10
Issue number5
DOIs
StatePublished - Jan 1 2018

Fingerprint

Intracranial Aneurysm
Aneurysm
Demography
Neurosurgeons
Population

Keywords

  • aneurysm
  • angiography
  • coil

ASJC Scopus subject areas

  • Surgery
  • Clinical Neurology

Cite this

Evaluation of previously embolized intracranial aneurysms : Inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists. / Zuckerman, Scott L.; Lakomkin, Nikita; Magarik, Jordan A.; Vargas, Jan; Stephens, Marcus; Akinpelu, Babatunde; Spiotta, Alejandro M.; Ahmed, Azam; Arthur, Adam S.; Fiorella, David; Hanel, Ricardo; Hirsch, Joshua A.; Hui, Ferdinand; James, Robert F.; Kallmes, David F.; Meyers, Philip M.; Niemann, David B.; Rasmussen, Peter; Turner, Raymond D.; Welch, Babu G.; Mocco, J.

In: Journal of NeuroInterventional Surgery, Vol. 10, No. 5, 01.01.2018, p. 464-468.

Research output: Contribution to journalArticle

Zuckerman, SL, Lakomkin, N, Magarik, JA, Vargas, J, Stephens, M, Akinpelu, B, Spiotta, AM, Ahmed, A, Arthur, AS, Fiorella, D, Hanel, R, Hirsch, JA, Hui, F, James, RF, Kallmes, DF, Meyers, PM, Niemann, DB, Rasmussen, P, Turner, RD, Welch, BG & Mocco, J 2018, 'Evaluation of previously embolized intracranial aneurysms: Inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists', Journal of NeuroInterventional Surgery, vol. 10, no. 5, pp. 464-468. https://doi.org/10.1136/neurintsurg-2017-013231
Zuckerman, Scott L. ; Lakomkin, Nikita ; Magarik, Jordan A. ; Vargas, Jan ; Stephens, Marcus ; Akinpelu, Babatunde ; Spiotta, Alejandro M. ; Ahmed, Azam ; Arthur, Adam S. ; Fiorella, David ; Hanel, Ricardo ; Hirsch, Joshua A. ; Hui, Ferdinand ; James, Robert F. ; Kallmes, David F. ; Meyers, Philip M. ; Niemann, David B. ; Rasmussen, Peter ; Turner, Raymond D. ; Welch, Babu G. ; Mocco, J. / Evaluation of previously embolized intracranial aneurysms : Inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists. In: Journal of NeuroInterventional Surgery. 2018 ; Vol. 10, No. 5. pp. 464-468.
@article{ffd92406d3aa4847a13493233e7bd9c1,
title = "Evaluation of previously embolized intracranial aneurysms: Inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists",
abstract = "Background The angiographic evaluation of previously coiled aneurysms can be difficult yet remains critical for determining re-treatment. Objective The main objective of this study was to determine the inter-rater reliability for both the Raymond Scale and per cent embolization among a group of neurointerventionalists evaluating previously embolized aneurysms. Methods A panel of 15 neurointerventionalists examined 92 distinct cases of immediate post-coil embolization and 1 year post-embolization angiographs. Each case was presented four times throughout the study, along with alterations in demographics in order to evaluate intra-rater reliability. All respondents were asked to provide the per cent embolization (0-100{\%}) and Raymond Scale grade (1-3) for each aneurysm. Inter-rater reliability was evaluated by computing weighted kappa values (for the Raymond Scale) and intraclass correlation coefficients (ICC) for per cent embolization. Results 10 neurosurgeons and 5 interventional neuroradiologists evaluated 368 simulated cases. The agreement among all readers employing the Raymond Scale was fair (κ =0.35) while concordance in per cent embolization was good (ICC=0.64). Clinicians with fewer than 10 years of experience demonstrated a significantly greater level of agreement than the group with greater than 10 years (κ =0.39 and ICC=0.70 vs κ =0.28 and ICC=0.58). When the same aneurysm was presented multiple times, clinicians demonstrated excellent consistency when assessing per cent embolization (ICC=0.82), but moderate agreement when employing the Raymond classification (κ =0.58). Conclusions Identifying the per cent embolization in previously coiled aneurysms resulted in good inter- and intra-rater agreement, regardless of years of experience. The strong agreement among providers employing per cent embolization may make it a valuable tool for embolization assessment in this patient population.",
keywords = "aneurysm, angiography, coil",
author = "Zuckerman, {Scott L.} and Nikita Lakomkin and Magarik, {Jordan A.} and Jan Vargas and Marcus Stephens and Babatunde Akinpelu and Spiotta, {Alejandro M.} and Azam Ahmed and Arthur, {Adam S.} and David Fiorella and Ricardo Hanel and Hirsch, {Joshua A.} and Ferdinand Hui and James, {Robert F.} and Kallmes, {David F.} and Meyers, {Philip M.} and Niemann, {David B.} and Peter Rasmussen and Turner, {Raymond D.} and Welch, {Babu G.} and J. Mocco",
year = "2018",
month = "1",
day = "1",
doi = "10.1136/neurintsurg-2017-013231",
language = "English (US)",
volume = "10",
pages = "464--468",
journal = "Journal of NeuroInterventional Surgery",
issn = "1759-8478",
publisher = "BMJ Publishing Group",
number = "5",

}

TY - JOUR

T1 - Evaluation of previously embolized intracranial aneurysms

T2 - Inter-and intra-rater reliability among neurosurgeons and interventional neuroradiologists

AU - Zuckerman, Scott L.

AU - Lakomkin, Nikita

AU - Magarik, Jordan A.

AU - Vargas, Jan

AU - Stephens, Marcus

AU - Akinpelu, Babatunde

AU - Spiotta, Alejandro M.

AU - Ahmed, Azam

AU - Arthur, Adam S.

AU - Fiorella, David

AU - Hanel, Ricardo

AU - Hirsch, Joshua A.

AU - Hui, Ferdinand

AU - James, Robert F.

AU - Kallmes, David F.

AU - Meyers, Philip M.

AU - Niemann, David B.

AU - Rasmussen, Peter

AU - Turner, Raymond D.

AU - Welch, Babu G.

AU - Mocco, J.

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Background The angiographic evaluation of previously coiled aneurysms can be difficult yet remains critical for determining re-treatment. Objective The main objective of this study was to determine the inter-rater reliability for both the Raymond Scale and per cent embolization among a group of neurointerventionalists evaluating previously embolized aneurysms. Methods A panel of 15 neurointerventionalists examined 92 distinct cases of immediate post-coil embolization and 1 year post-embolization angiographs. Each case was presented four times throughout the study, along with alterations in demographics in order to evaluate intra-rater reliability. All respondents were asked to provide the per cent embolization (0-100%) and Raymond Scale grade (1-3) for each aneurysm. Inter-rater reliability was evaluated by computing weighted kappa values (for the Raymond Scale) and intraclass correlation coefficients (ICC) for per cent embolization. Results 10 neurosurgeons and 5 interventional neuroradiologists evaluated 368 simulated cases. The agreement among all readers employing the Raymond Scale was fair (κ =0.35) while concordance in per cent embolization was good (ICC=0.64). Clinicians with fewer than 10 years of experience demonstrated a significantly greater level of agreement than the group with greater than 10 years (κ =0.39 and ICC=0.70 vs κ =0.28 and ICC=0.58). When the same aneurysm was presented multiple times, clinicians demonstrated excellent consistency when assessing per cent embolization (ICC=0.82), but moderate agreement when employing the Raymond classification (κ =0.58). Conclusions Identifying the per cent embolization in previously coiled aneurysms resulted in good inter- and intra-rater agreement, regardless of years of experience. The strong agreement among providers employing per cent embolization may make it a valuable tool for embolization assessment in this patient population.

AB - Background The angiographic evaluation of previously coiled aneurysms can be difficult yet remains critical for determining re-treatment. Objective The main objective of this study was to determine the inter-rater reliability for both the Raymond Scale and per cent embolization among a group of neurointerventionalists evaluating previously embolized aneurysms. Methods A panel of 15 neurointerventionalists examined 92 distinct cases of immediate post-coil embolization and 1 year post-embolization angiographs. Each case was presented four times throughout the study, along with alterations in demographics in order to evaluate intra-rater reliability. All respondents were asked to provide the per cent embolization (0-100%) and Raymond Scale grade (1-3) for each aneurysm. Inter-rater reliability was evaluated by computing weighted kappa values (for the Raymond Scale) and intraclass correlation coefficients (ICC) for per cent embolization. Results 10 neurosurgeons and 5 interventional neuroradiologists evaluated 368 simulated cases. The agreement among all readers employing the Raymond Scale was fair (κ =0.35) while concordance in per cent embolization was good (ICC=0.64). Clinicians with fewer than 10 years of experience demonstrated a significantly greater level of agreement than the group with greater than 10 years (κ =0.39 and ICC=0.70 vs κ =0.28 and ICC=0.58). When the same aneurysm was presented multiple times, clinicians demonstrated excellent consistency when assessing per cent embolization (ICC=0.82), but moderate agreement when employing the Raymond classification (κ =0.58). Conclusions Identifying the per cent embolization in previously coiled aneurysms resulted in good inter- and intra-rater agreement, regardless of years of experience. The strong agreement among providers employing per cent embolization may make it a valuable tool for embolization assessment in this patient population.

KW - aneurysm

KW - angiography

KW - coil

UR - http://www.scopus.com/inward/record.url?scp=85046424701&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85046424701&partnerID=8YFLogxK

U2 - 10.1136/neurintsurg-2017-013231

DO - 10.1136/neurintsurg-2017-013231

M3 - Article

C2 - 28918386

AN - SCOPUS:85046424701

VL - 10

SP - 464

EP - 468

JO - Journal of NeuroInterventional Surgery

JF - Journal of NeuroInterventional Surgery

SN - 1759-8478

IS - 5

ER -