CNN-based hierarchical coarse-to-fine segmentation of pelvic CT images for prostate cancer radiotherapy

Sharmin Sultana, Adam Robinson, Daniel Y. Song, Junghoon Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Accurate segmentation of organs-at-risk is important in prostate cancer radiation therapy planning. However, poor soft tissue contrast in CT makes the segmentation task very challenging. We propose a deep convolutional neural network approach to automatically segment the prostate, bladder, and rectum from pelvic CT. A hierarchical coarse-to-fine segmentation strategy is used where the first step generates a coarse segmentation from which an organ-specific region of interest (ROI) localization map is produced. The second step produces detailed and accurate segmentation of the organs. The ROI localization map is generated using a 3D U-net. The localization map helps adjusting the ROI of each organ that needs to be segmented and hence improves computational efficiency by eliminating irrelevant background information. For the fine segmentation step, we designed a fully convolutional network (FCN) by combining a generative adversarial network (GAN) with a U-net. Specifically, the generator is a 3D U-net that is trained to predict individual pelvic structures, and the discriminator is an FCN which fine-tunes the generator predicted segmentation map by comparing it with the ground truth. The network was trained using 100 CT datasets and tested on 15 datasets to segment the prostate, bladder and rectum. The average Dice similarity (mean±SD) of the prostate, bladder and rectum are 0.90±0.05, 0.96±0.06 and 0.91±0.09, respectively, and Hausdorff distances of these three structures are 5.21±1.17, 4.37±0.56 and 6.11±1.47 (mm), respectively. The proposed method produces accurate and reproducible segmentation of pelvic structures, which can be potentially valuable for prostate cancer radiotherapy treatment planning.

Original languageEnglish (US)
Title of host publicationMedical Imaging 2020
Subtitle of host publicationImage-Guided Procedures, Robotic Interventions, and Modeling
EditorsBaowei Fei, Cristian A. Linte
ISBN (Electronic)9781510633971
StatePublished - 2020
EventMedical Imaging 2020: Image-Guided Procedures, Robotic Interventions, and Modeling - Houston, United States
Duration: Feb 16 2020Feb 19 2020

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X


ConferenceMedical Imaging 2020: Image-Guided Procedures, Robotic Interventions, and Modeling
Country/TerritoryUnited States


  • CT image
  • Hierarchical segmentation
  • Male pelvic organ
  • Prostate cancer
  • Radiotherapy

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'CNN-based hierarchical coarse-to-fine segmentation of pelvic CT images for prostate cancer radiotherapy'. Together they form a unique fingerprint.

Cite this