Fusion of concurrent single trial EEG data and FMRI data using multi-set canonical correlation analysis

We propose a data fusion method for the fusion of simultaneously acquired functional magnetic resonance imaging (fMRI) and single trial electroencephalography (EEG) data from multiple subjects using multi-set canonical correlation analysis (M-CCA). Our proposed technique utilizes the common time series information in the multimodal datasets to find trial-to-trial covariations across modalities, and based on these covariations, the data is decomposed into spatial maps for the fMRI data and a corresponding temporal evolution for the EEG data. Additionally, the analysis is performed simultaneously on data from a group of subjects, thus providing an efficient tool to make group inferences about cross-modality covariation. The proposed method is multivariate and hence facilitates the study of brain connectivity along with localization of brain function. We demonstrate the promise of the method in finding covarying trial-to-trial amplitude modulations in an auditory task involving implicit pattern learning.