TY - JOUR
T1 - Recurrent Saliency Transformation Network for Tiny Target Segmentation in Abdominal CT Scans
AU - Xie, Lingxi
AU - Yu, Qihang
AU - Zhou, Yuyin
AU - Wang, Yan
AU - Fishman, Elliot K.
AU - Yuille, Alan L.
N1 - Funding Information:
Manuscript received May 5, 2019; revised July 4, 2019; accepted July 16, 2019. Date of publication July 23, 2019; date of current version January 31, 2020. This work was supported by the Lustgarten Foundation for Pancreatic Cancer Research. (Lingxi Xie and Qihang Yu contributed equally to this work.) (Corresponding author: Yan Wang.) L. Xie was with the Department of Computer Science, Johns Hopkins University, Baltimore, MD 21218 USA. He is now with the Huawei Noah’s Ark Lab, Beijing 100085, China (e-mail: 198808xc@gmail.com).
Publisher Copyright:
© 1982-2012 IEEE.
PY - 2020/2
Y1 - 2020/2
N2 - We aim at segmenting a wide variety of organs, including tiny targets (e.g., adrenal gland), and neoplasms (e.g., pancreatic cyst), from abdominal CT scans. This is a challenging task in two aspects. First, some organs (e.g., the pancreas), are highly variable in both anatomy and geometry, and thus very difficult to depict. Second, the neoplasms often vary a lot in its size, shape, as well as its location within the organ. Third, the targets (organs and neoplasms) can be considerably small compared to the human body, and so standard deep networks for segmentation are often less sensitive to these targets and thus predict less accurately especially around their boundaries. In this paper, we present an end-to-end framework named recurrent saliency transformation network (RSTN) for segmenting tiny and/or variable targets. The RSTN is a coarse-to-fine approach that uses prediction from the first (coarse) stage to shrink the input region for the second (fine) stage. A saliency transformation module is inserted between these two stages so that 1) the coarse-scaled segmentation mask can be transferred as spatial weights and applied to the fine stage and 2) the gradients can be back-propagated from the loss layer to the entire network so that the two stages are optimized in a joint manner. In the testing stage, we perform segmentation iteratively to improve accuracy. In this extended journal paper, we allow a gradual optimization to improve the stability of the RSTN, and introduce a hierarchical version named H-RSTN to segment tiny and variable neoplasms such as pancreatic cysts. Experiments are performed on several CT datasets including a public pancreas segmentation dataset, our own multi-organ dataset, and a cystic pancreas dataset. In all these cases, the RSTN outperforms the baseline (a stage-wise coarse-to-fine approach) significantly. Confirmed by the radiologists in our team, these promising segmentation results can help early diagnosis of pancreatic cancer. The code and pre-trained models of our project were made available at https://github.com/198808xc/OrganSegRSTN.
AB - We aim at segmenting a wide variety of organs, including tiny targets (e.g., adrenal gland), and neoplasms (e.g., pancreatic cyst), from abdominal CT scans. This is a challenging task in two aspects. First, some organs (e.g., the pancreas), are highly variable in both anatomy and geometry, and thus very difficult to depict. Second, the neoplasms often vary a lot in its size, shape, as well as its location within the organ. Third, the targets (organs and neoplasms) can be considerably small compared to the human body, and so standard deep networks for segmentation are often less sensitive to these targets and thus predict less accurately especially around their boundaries. In this paper, we present an end-to-end framework named recurrent saliency transformation network (RSTN) for segmenting tiny and/or variable targets. The RSTN is a coarse-to-fine approach that uses prediction from the first (coarse) stage to shrink the input region for the second (fine) stage. A saliency transformation module is inserted between these two stages so that 1) the coarse-scaled segmentation mask can be transferred as spatial weights and applied to the fine stage and 2) the gradients can be back-propagated from the loss layer to the entire network so that the two stages are optimized in a joint manner. In the testing stage, we perform segmentation iteratively to improve accuracy. In this extended journal paper, we allow a gradual optimization to improve the stability of the RSTN, and introduce a hierarchical version named H-RSTN to segment tiny and variable neoplasms such as pancreatic cysts. Experiments are performed on several CT datasets including a public pancreas segmentation dataset, our own multi-organ dataset, and a cystic pancreas dataset. In all these cases, the RSTN outperforms the baseline (a stage-wise coarse-to-fine approach) significantly. Confirmed by the radiologists in our team, these promising segmentation results can help early diagnosis of pancreatic cancer. The code and pre-trained models of our project were made available at https://github.com/198808xc/OrganSegRSTN.
KW - Semantic segmentation
KW - abdominal CT scan
KW - coarse-to-fine
KW - deep neural network
KW - saliency transformation
UR - http://www.scopus.com/inward/record.url?scp=85079020265&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85079020265&partnerID=8YFLogxK
U2 - 10.1109/TMI.2019.2930679
DO - 10.1109/TMI.2019.2930679
M3 - Article
C2 - 31352338
AN - SCOPUS:85079020265
SN - 0278-0062
VL - 39
SP - 514
EP - 525
JO - IEEE transactions on medical imaging
JF - IEEE transactions on medical imaging
IS - 2
M1 - 8769868
ER -