Segmentation of optical coherence tomography images for differentiation of the cavernous nerves from the prostate gland

The cavernous nerves course along the surface of the prostate and are responsible for erectile function. Improvements in identification, imaging, and visualization of the cavernous nerves during prostate cancer surgery may improve nerve preservation and postoperative sexual potency. Two-dimensional (2-D) optical coherence tomography (OCT) images of the rat prostate were segmented to differentiate the cavernous nerves from the prostate gland. To detect these nerves, three image features were employed: Gabor filter, Daubechies wavelet, and Laws filter. The Gabor feature was applied with different standard deviations in the x and y directions. In the Daubechies wavelet feature, an 8-tap Daubechies orthonormal wavelet was implemented, and the low-pass sub-band was chosen as the filtered image. Last, Laws feature extraction was applied to the images. The features were segmented using a nearest-neighbor classifier. N-ary morphological postprocessing was used to remove small voids. The cavernous nerves were differentiated from the prostate gland with a segmentation error rate of only 0.058±0.019. This algorithm may be useful for implementation in clinical endoscopic OCT systems currently being studied for potential intraoperative diagnostic use in laparoscopic and robotic nerve-sparing prostate cancer surgery.