A deep-learning approach to translate between brain structure and functional connectivity

Vince D. Calhoun, Md Faijul Amin, Devon Hjelm, Eswar Damaraju, Sergey M. Plis

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

While the majority of exploratory approaches search for correlations among features of different modalities, indirect/nonlinear relations between structure and function have not yet been fully investigated. In this work, we employ a neural machine translation model [1] to relate two modalities: structural MRI (sMRI) spatial components and functional MRI (fMRI) brain states estimated using a dynamic connectivity model. We consider each of the modalities as different 'languages' of the same brain and fit a translation model to estimate a model for how structure influences function. Results identify multiple aligned aspects of brain structure and functional brain states showing significantly more or less alignment in the patient group as well as interesting links to other variables such as cognitive scores and symptom assessments. Our novel approach provides a new perspective on combining brain structure and function by incorporating indirect/nonlinear effects and enabling the algorithm to learn the interplay between structural and the functional networks.

Original languageEnglish (US)
Title of host publication2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages6155-6159
Number of pages5
ISBN (Electronic)9781509041176
DOIs
StatePublished - Jun 16 2017
Event2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017 - New Orleans, United States
Duration: Mar 5 2017Mar 9 2017

Publication series

NameICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
ISSN (Print)1520-6149

Other

Other2017 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2017
Country/TerritoryUnited States
CityNew Orleans
Period3/5/173/9/17

Keywords

  • deep learning
  • multimodal fusion
  • psychosis
  • schizophrenia

ASJC Scopus subject areas

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'A deep-learning approach to translate between brain structure and functional connectivity'. Together they form a unique fingerprint.

Cite this