Multivariate Machine Learning Methods for Fusing Multimodal Functional Neuroimaging Data

Sven Dahne, Felix Bieszmann, Wojciech Samek, Stefan Haufe, Dominique Goltz, Christopher Gundlach, Arno Villringer, Siamac Fazli, Klaus Robert Muller

Research output: Contribution to journalArticlepeer-review

74 Citations (Scopus)


Multimodal data are ubiquitous in engineering, communications, robotics, computer vision, or more generally speaking in industry and the sciences. All disciplines have developed their respective sets of analytic tools to fuse the information that is available in all measured modalities. In this paper, we provide a review of classical as well as recent machine learning methods (specifically factor models) for fusing information from functional neuroimaging techniques such as: LFP, EEG, MEG, fNIRS, and fMRI. Early and late fusion scenarios are distinguished, and appropriate factor models for the respective scenarios are presented along with example applications from selected multimodal neuroimaging studies. Further emphasis is given to the interpretability of the resulting model parameters, in particular by highlighting how factor models relate to physical models needed for source localization. The methods we discuss allow for the extraction of information from neural data, which ultimately contributes to 1) better neuroscientific understanding; 2) enhance diagnostic performance; and 3) discover neural signals of interest that correlate maximally with a given cognitive paradigm. While we clearly study the multimodal functional neuroimaging challenge, the discussed machine learning techniques have a wide applicability, i.e., in general data fusion, and may thus be informative to the general interested reader.

Original languageEnglish
Article number7182735
Pages (from-to)1507-1530
Number of pages24
JournalProceedings of the IEEE
Issue number9
Publication statusPublished - 2015 Sept 1

Bibliographical note

Publisher Copyright:
© 2015 IEEE.


  • EEG
  • MEG
  • Machine learning
  • data fusion
  • fMRI
  • fNIRS
  • multimodal neuroimaging
  • review

ASJC Scopus subject areas

  • Electrical and Electronic Engineering


Dive into the research topics of 'Multivariate Machine Learning Methods for Fusing Multimodal Functional Neuroimaging Data'. Together they form a unique fingerprint.

Cite this