Performance assessment framework for neural network denoising

The proliferation of deep learning image processing calls for a quantitative image quality assessment framework that is suitable for nonlinear, data-dependent algorithms. In this work, we propose a method to systematically evaluate the system and noise responses such that the nonlinear transfer properties can be mapped out. The method involves sampling of lesion perturbations as a function of size, contrast, as well as clinically relevant features such as shape and texture that may be important for diagnosis. We embed the perturbations in backgrounds of varying attenuation levels, noise magnitude and correlation that are associated with different patient anatomies and imaging protocols. The range of system and noise response are further used to evaluate performance for clinical tasks such as signal detection and classification. We performed the assessment for an example CNN-denoising algorithm for low does lung CT screening. The system response of the CNN-denoising algorithm exhibits highly nonlinear behavior where both contrast and higher order lesion features such as spiculated boundaries are not reliably represented for lesions perturbations with small size and low contrast. The noise properties are potentially highly nonstationary, and should be assumed to be the same between the signal-present and signal-absent images. Furthermore, we observer a high degree dependency of both system and noise response on the background attenuation levels. Inputs around zeros are effectively imposed a non-negativity constraint; transfer properties for higher background levels are highly variable. For a detection task, CNN-denoised images improved detectability index by 16-18% compared to low dose CT inputs. For classification task between spiculated and smooth lesions, CNN-denoised images result in a much larger improvement up to 50%. The performance assessment framework propose in this work can systematically map out the nonlinear transfer functions for deep learning algorithms and can potentially enable robust deployment of such algorithms in medical imaging applications.