A Unified Multi-Modality Fusion Framework for Deep Spatio-Temporal-Spectral Feature Learning in Resting-State fMRI Denoising

Minjoo Lim, Keun Soo Heo, Jun Mo Kim, Bogyeong Kang, Weili Lin, Han Zhang, Dinggang Shen, Tae Eui Kam

Research output: Contribution to journalArticlepeer-review


Resting-state functional magnetic resonance imaging (rs-fMRI) is a commonly used functional neuroimaging technique to investigate the functional brain networks. However, rs-fMRI data are often contaminated with noise and artifacts that adversely affect the results of rs-fMRI studies. Several machine/deep learning methods have achieved impressive performance to automatically regress the noise-related components decomposed from rs-fMRI data, which are expressed as the pairs of a spatial map and its associated time series. However, most of the previous methods individually analyze each modality of the noise-related components and simply aggregate the decision-level information (or knowledge) extracted from each modality to make a final decision. Moreover, these approaches consider only the limited modalities making it difficult to explore class-discriminative spectral information of noise-related components. To overcome these limitations, we propose a unified deep attentive spatio-spectral-temporal feature fusion framework. We first adopt a learnable wavelet transform module at the input-level of the framework to elaborately explore the spectral information in subsequent processes. We then construct a feature-level multi-modality fusion module to efficiently exchange the information from multi-modality inputs in the feature space. Finally, we design confidence-based voting strategies for decision-level fusion at the end of the framework to make a robust final decision. In our experiments, the proposed method achieved remarkable performance for noise-related component detection on various rs-fMRI datasets.

Original languageEnglish
Pages (from-to)1-12
Number of pages12
JournalIEEE Journal of Biomedical and Health Informatics
Publication statusAccepted/In press - 2024

Bibliographical note

Publisher Copyright:


  • Convolutional neural network
  • Deep learning
  • Denoising
  • Feature extraction
  • Integrated circuit modeling
  • Kernel
  • Multi-modality fusion
  • Noise reduction
  • Resting-state fMRI
  • Streams
  • Three-dimensional displays
  • Time series analysis

ASJC Scopus subject areas

  • Computer Science Applications
  • Health Informatics
  • Electrical and Electronic Engineering
  • Health Information Management


Dive into the research topics of 'A Unified Multi-Modality Fusion Framework for Deep Spatio-Temporal-Spectral Feature Learning in Resting-State fMRI Denoising'. Together they form a unique fingerprint.

Cite this