Efficient test for nonlinear dependence of two continuous variables

Background: Testing dependence/correlation of two variables is one of the fundamental tasks in statistics. In this work, we proposed a new way of testing nonlinear dependence between two continuous variables (X and Y). Results: We addressed this research question by using CANOVA (continuous analysis of variance, software available at https://sourceforge.net/projects/canova/ ). In the CANOVA framework, we first defined a neighborhood for each data point related to its X value, and then calculated the variance of the Y value within the neighborhood. Finally, we performed permutations to evaluate the significance of the observed values within the neighborhood variance. To evaluate the strength of CANOVA compared to six other methods, we performed extensive simulations to explore the relationship between methods and compared the false positive rates and statistical power using both simulated and real datasets (kidney cancer RNA-seq dataset). Conclusions: We concluded that CANOVA is an efficient method for testing nonlinear correlation with several advantages in real data applications.