Effective fusion of structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) data has the potential to boost the accuracy of infant age prediction thanks to the complementary information provided by different imaging modalities. However, functional connectivity measured by fMRI during infancy is largely immature and noisy compared to the morphological features from sMRI, thus making the sMRI and fMRI fusion for infant brain analysis extremely challenging. With the conventional multimodal fusion strategies, adding fMRI data for age prediction has a high risk of introducing more noises than useful features, which would lead to reduced accuracy than that merely using sMRI data. To address this issue, we develop a novel model termed as disentangled-multimodal adversarial autoencoder (DMM-AAE) for infant age prediction based on multimodal brain MRI. Specifically, we disentangle the latent variables of autoencoder into common and specific codes to represent the shared and complementary information among modalities, respectively. Then, cross-reconstruction requirement and common-specific distance ratio loss are designed as regularizations to ensure the effectiveness and thoroughness of the disentanglement. By arranging relatively independent autoencoders to separate the modalities and employing disentanglement under cross-reconstruction requirement to integrate them, our DMM-AAE method effectively restrains the possible interference cross modalities, while realizing effective information fusion. Taking advantage of the latent variable disentanglement, a new strategy is further proposed and embedded into DMM-AAE to address the issue of incompleteness of the multimodal neuroimages, which can also be used as an independent algorithm for missing modality imputation. By taking six types of cortical morphometric features from sMRI and brain functional connectivity from fMRI as predictors, the superiority of the proposed DMM-AAE is validated on infant age (35 to 848 days after birth) prediction using incomplete multimodal neuroimages. The mean absolute error of the prediction based on DMM-AAE reaches 37.6 days, outperforming state-of-the-art methods. Generally, our proposed DMM-AAE can serve as a promising model for prediction with multimodal data.
Bibliographical noteFunding Information:
This work was supported in part by NIH under Grant MH116225, Grant MH117943, Grant MH104324, Grant MH109773, and Grant 1U01MH110274 and in part by the Efforts of the UNC/UMN Baby Connectome Project Consortium.
specific codes to represent the shared and complementary N I. INTRODUCTION information among modalities, respectively. Then, cross- EUROIMAGING-BASED age prediction is important ratio lossare designedasregularizationsto ensure thereconstructionrequirementandcommon-specificdistance for brain development analysis and early detection of effectiveness and thoroughness of the disentanglement. neurodevelopmental disorders . The discrepancy between By arranging relatively independent autoencoders to the “chronological age” and the predicted “brain age” can separate the modalities and employing disentanglement be considered as an index of deviation from the normative undercross-reconstructionrequirementtointegratethem, developmental or aging trajectory. For example, the predicted our DMM-AAE method effectively restrains the possible age with neuroimaging data can be used to detect accelerated atrophy after traumatic brain injury  and accelerated brain July26,2020.DateofpublicationAugust3,2020;dateofcurrentversionManuscriptreceivedMay22,2020;revisedJuly20,2020;accepted aging due to schizophrenia , type-2 diabetes mellitus , November 30, 2020. This work was supported in part by NIH under Grant and HIV disease . Furthermore, prediction of brain age is MH116225,GrantMH117943,GrantMH104324,GrantMH109773,and also used to help discern possible environmental and lifestyle BabyConnectome ProjectConsortium.(Correspondingauthors:Grant1U01MH110274and inpart bytheEfforts of the UNC/UMN related influences on the human brain, e.g., younger brain Gang Li; Dinggang Shen.) age due to higher education, more self-reported physical Dan Hu, Han Zhang, Zhengwang Wu, Fan Wang, Li Wang, activity , and long-term meditation practice , as well as Radiology,Universityof NorthCarolinaat ChapelHill,ChapelHill,J.KeithSmith,WeiliLin,andGangLi arewith theDepartmentof increased brain age associated with midlife . NC 27599 USA (e-mail: email@example.com; firstname.lastname@example.org; Since different modalities of neuroimages can provide email@example.com; firstname.lastname@example.org; li_wang@ plementary information to each other, researchers started to email@example.com).med.unc.edu;firstname.lastname@example.org; email@example.com; combine multimodal imaging to predict brain age. For exam- Dinggang Shen is with the Department of Radiology, University of ple, to capture cognitive impairment, functional connectivity NorthCarolinaatChapelHill,ChapelHill,NC27599USA,andalsowith derived from rs-fMRI and cortical morphological features SouthKorea(e-mail:firstname.lastname@example.org).theDepartmentofArtificialIntelligence,KoreaUniversity,Seoul02841, from sMRI were combined for brain age prediction . This article has supplementary downloadable material available at In , T2-weighted, T1-weighted, diffusion-weighted, and https://ieeexplore.ieee.org,providedbytheauthors. fluid-attenuate diversion recovery (FLAIR) scans were com-onlineathttps://ieeexplore.ieee.org.Colorversionsofoneormoreofthefiguresinthisarticleareavailable bined for age prediction, which highlighted the importance of Digital Object Identifier 10.1109/TMI.2020.3013825 using multimodal biomarkers to study normal aging.
© 1982-2012 IEEE.
- Infant age prediction
- magnetic resonance imaging
- multimodal machine learning
ASJC Scopus subject areas
- Radiological and Ultrasound Technology
- Computer Science Applications
- Electrical and Electronic Engineering