Can AI outperform a junior resident? Comparison of deep neural network to first-year radiology residents for identification of pneumothorax

Paul H. Yi, Tae Kyung Kim, Alice C. Yu, Bradford Bennett, John Eng, Cheng Ting Lin

Research output: Contribution to journalArticle

Abstract

Purpose: To (1) develop a deep learning system (DLS) using a deep convolutional neural network (DCNN) for identification of pneumothorax, (2) compare its performance to first-year radiology residents, and (3) evaluate the ability of a DLS to augment radiology residents by detecting missed pneumothoraces. Methods: This was a retrospective study performed in September 2018. We obtained 112,120 chest radiographs (CXRs) from the NIH ChestXray14 database, of which 4360 cases (4%) were labeled as pneumothorax by natural language processing. We utilized 111,518 CXRs to train and validate the ResNet-152 DCNN pretrained on ImageNet to identify pneumothorax. DCNN testing was performed on a hold-out set of 602 CXRs, whose groundtruth was determined by a cardiothoracic radiologist. Two first-year radiology residents evaluated the test CXRs for presence of pneumothorax. Receiver operating characteristic (ROC) curves were generated for each evaluator with area under the curve (AUC) compared using the DeLong parametric method. Results: The DCNN achieved AUC of 0.841 for identification of pneumothorax at a rate of 1980 images/min. In contrast, both first-year residents achieved significantly higher AUCs of 0.942 and 0.905 (p < 0.01 for both compared to DCNN), but at a slower rate of two images/min. The DCNN identified 3 of 31 (9.7%) additional pneumothoraces missed by at least one of the residents. Conclusion: A DLS for pneumothorax identification had lower AUC than 1st-year radiology residents, but interpreted images > 1000× as fast and identified 3 additional pneumothoraces missed by the residents. Our findings suggest that DLS could augment radiologists-in-training to identify potential urgent findings.

Original languageEnglish (US)
Pages (from-to)367-375
Number of pages9
JournalEmergency Radiology
Volume27
Issue number4
DOIs
StatePublished - Aug 1 2020

Keywords

  • Artificial intelligence
  • Chest radiograph
  • Deep learning
  • Machine learning
  • Radiology residents

ASJC Scopus subject areas

  • Emergency Medicine
  • Radiology Nuclear Medicine and imaging

Fingerprint Dive into the research topics of 'Can AI outperform a junior resident? Comparison of deep neural network to first-year radiology residents for identification of pneumothorax'. Together they form a unique fingerprint.

  • Cite this