Disentangled-Multimodal Adversarial Autoencoder: Application to Infant Age Prediction with Incomplete Multimodal Neuroimages

Dan Hu; Han Zhang; Zhengwang Wu; Fan Wang; Li Wang; J. Keith Smith; Weili Lin; Gang Li; Dinggang Shen

doi:10.1109/TMI.2020.3013825

Disentangled-Multimodal Adversarial Autoencoder: Application to Infant Age Prediction with Incomplete Multimodal Neuroimages

Dan Hu, Han Zhang, Zhengwang Wu, Fan Wang, Li Wang, J. Keith Smith, Weili Lin, Gang Li, Dinggang Shen

Department of Brain and Cognitive Engineering

Research output: Contribution to journal › Article › peer-review

23 Citations (Scopus)

Abstract

Effective fusion of structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) data has the potential to boost the accuracy of infant age prediction thanks to the complementary information provided by different imaging modalities. However, functional connectivity measured by fMRI during infancy is largely immature and noisy compared to the morphological features from sMRI, thus making the sMRI and fMRI fusion for infant brain analysis extremely challenging. With the conventional multimodal fusion strategies, adding fMRI data for age prediction has a high risk of introducing more noises than useful features, which would lead to reduced accuracy than that merely using sMRI data. To address this issue, we develop a novel model termed as disentangled-multimodal adversarial autoencoder (DMM-AAE) for infant age prediction based on multimodal brain MRI. Specifically, we disentangle the latent variables of autoencoder into common and specific codes to represent the shared and complementary information among modalities, respectively. Then, cross-reconstruction requirement and common-specific distance ratio loss are designed as regularizations to ensure the effectiveness and thoroughness of the disentanglement. By arranging relatively independent autoencoders to separate the modalities and employing disentanglement under cross-reconstruction requirement to integrate them, our DMM-AAE method effectively restrains the possible interference cross modalities, while realizing effective information fusion. Taking advantage of the latent variable disentanglement, a new strategy is further proposed and embedded into DMM-AAE to address the issue of incompleteness of the multimodal neuroimages, which can also be used as an independent algorithm for missing modality imputation. By taking six types of cortical morphometric features from sMRI and brain functional connectivity from fMRI as predictors, the superiority of the proposed DMM-AAE is validated on infant age (35 to 848 days after birth) prediction using incomplete multimodal neuroimages. The mean absolute error of the prediction based on DMM-AAE reaches 37.6 days, outperforming state-of-the-art methods. Generally, our proposed DMM-AAE can serve as a promising model for prediction with multimodal data.

Original language	English
Article number	9154745
Pages (from-to)	4137-4149
Number of pages	13
Journal	IEEE Transactions on Medical Imaging
Volume	39
Issue number	12
DOIs	https://doi.org/10.1109/TMI.2020.3013825
Publication status	Published - 2020 Dec

Bibliographical note

Funding Information:
This work was supported in part by NIH under Grant MH116225, Grant MH117943, Grant MH104324, Grant MH109773, and Grant 1U01MH110274 and in part by the Efforts of the UNC/UMN Baby Connectome Project Consortium.

Funding Information:
specific codes to represent the shared and complementary N I. INTRODUCTION information among modalities, respectively. Then, cross- EUROIMAGING-BASED age prediction is important ratio lossare designedasregularizationsto ensure thereconstructionrequirementandcommon-specificdistance for brain development analysis and early detection of effectiveness and thoroughness of the disentanglement. neurodevelopmental disorders [1]. The discrepancy between By arranging relatively independent autoencoders to the “chronological age” and the predicted “brain age” can separate the modalities and employing disentanglement be considered as an index of deviation from the normative undercross-reconstructionrequirementtointegratethem, developmental or aging trajectory. For example, the predicted our DMM-AAE method effectively restrains the possible age with neuroimaging data can be used to detect accelerated atrophy after traumatic brain injury [2] and accelerated brain July26,2020.DateofpublicationAugust3,2020;dateofcurrentversionManuscriptreceivedMay22,2020;revisedJuly20,2020;accepted aging due to schizophrenia [3], type-2 diabetes mellitus [4], November 30, 2020. This work was supported in part by NIH under Grant and HIV disease [5]. Furthermore, prediction of brain age is MH116225,GrantMH117943,GrantMH104324,GrantMH109773,and also used to help discern possible environmental and lifestyle BabyConnectome ProjectConsortium.(Correspondingauthors:Grant1U01MH110274and inpart bytheEfforts of the UNC/UMN related influences on the human brain, e.g., younger brain Gang Li; Dinggang Shen.) age due to higher education, more self-reported physical Dan Hu, Han Zhang, Zhengwang Wu, Fan Wang, Li Wang, activity [6], and long-term meditation practice [7], as well as Radiology,Universityof NorthCarolinaat ChapelHill,ChapelHill,J.KeithSmith,WeiliLin,andGangLi arewith theDepartmentof increased brain age associated with midlife [8]. NC 27599 USA (e-mail: danhu@ad.unc.edu; hanzhang@med.unc.edu; Since different modalities of neuroimages can provide com-zhengwang_wu@med.unc.edu; fan_wang@med.unc.edu; li_wang@ plementary information to each other, researchers started to gang_li@med.unc.edu).med.unc.edu;jksmith@med.unc.edu; weili_lin@med.unc.edu; combine multimodal imaging to predict brain age. For exam- Dinggang Shen is with the Department of Radiology, University of ple, to capture cognitive impairment, functional connectivity NorthCarolinaatChapelHill,ChapelHill,NC27599USA,andalsowith derived from rs-fMRI and cortical morphological features SouthKorea(e-mail:dgshen@med.unc.edu).theDepartmentofArtificialIntelligence,KoreaUniversity,Seoul02841, from sMRI were combined for brain age prediction [9]. This article has supplementary downloadable material available at In [10], T2-weighted, T1-weighted, diffusion-weighted, and https://ieeexplore.ieee.org,providedbytheauthors. fluid-attenuate diversion recovery (FLAIR) scans were com-onlineathttps://ieeexplore.ieee.org.Colorversionsofoneormoreofthefiguresinthisarticleareavailable bined for age prediction, which highlighted the importance of Digital Object Identifier 10.1109/TMI.2020.3013825 using multimodal biomarkers to study normal aging.

Publisher Copyright:
© 1982-2012 IEEE.

Keywords

Infant age prediction
autoencoder
magnetic resonance imaging
multimodal machine learning

ASJC Scopus subject areas

Software
Radiological and Ultrasound Technology
Computer Science Applications
Electrical and Electronic Engineering

Access to Document

10.1109/TMI.2020.3013825

Cite this

@article{a2424687df9a48aea2b27e1518cafa21,

title = "Disentangled-Multimodal Adversarial Autoencoder: Application to Infant Age Prediction with Incomplete Multimodal Neuroimages",

abstract = "Effective fusion of structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) data has the potential to boost the accuracy of infant age prediction thanks to the complementary information provided by different imaging modalities. However, functional connectivity measured by fMRI during infancy is largely immature and noisy compared to the morphological features from sMRI, thus making the sMRI and fMRI fusion for infant brain analysis extremely challenging. With the conventional multimodal fusion strategies, adding fMRI data for age prediction has a high risk of introducing more noises than useful features, which would lead to reduced accuracy than that merely using sMRI data. To address this issue, we develop a novel model termed as disentangled-multimodal adversarial autoencoder (DMM-AAE) for infant age prediction based on multimodal brain MRI. Specifically, we disentangle the latent variables of autoencoder into common and specific codes to represent the shared and complementary information among modalities, respectively. Then, cross-reconstruction requirement and common-specific distance ratio loss are designed as regularizations to ensure the effectiveness and thoroughness of the disentanglement. By arranging relatively independent autoencoders to separate the modalities and employing disentanglement under cross-reconstruction requirement to integrate them, our DMM-AAE method effectively restrains the possible interference cross modalities, while realizing effective information fusion. Taking advantage of the latent variable disentanglement, a new strategy is further proposed and embedded into DMM-AAE to address the issue of incompleteness of the multimodal neuroimages, which can also be used as an independent algorithm for missing modality imputation. By taking six types of cortical morphometric features from sMRI and brain functional connectivity from fMRI as predictors, the superiority of the proposed DMM-AAE is validated on infant age (35 to 848 days after birth) prediction using incomplete multimodal neuroimages. The mean absolute error of the prediction based on DMM-AAE reaches 37.6 days, outperforming state-of-the-art methods. Generally, our proposed DMM-AAE can serve as a promising model for prediction with multimodal data.",

keywords = "Infant age prediction, autoencoder, magnetic resonance imaging, multimodal machine learning",

author = "Dan Hu and Han Zhang and Zhengwang Wu and Fan Wang and Li Wang and Smith, {J. Keith} and Weili Lin and Gang Li and Dinggang Shen",

note = "Funding Information: This work was supported in part by NIH under Grant MH116225, Grant MH117943, Grant MH104324, Grant MH109773, and Grant 1U01MH110274 and in part by the Efforts of the UNC/UMN Baby Connectome Project Consortium. Funding Information: specific codes to represent the shared and complementary N I. INTRODUCTION information among modalities, respectively. Then, cross- EUROIMAGING-BASED age prediction is important ratio lossare designedasregularizationsto ensure thereconstructionrequirementandcommon-specificdistance for brain development analysis and early detection of effectiveness and thoroughness of the disentanglement. neurodevelopmental disorders [1]. The discrepancy between By arranging relatively independent autoencoders to the “chronological age” and the predicted “brain age” can separate the modalities and employing disentanglement be considered as an index of deviation from the normative undercross-reconstructionrequirementtointegratethem, developmental or aging trajectory. For example, the predicted our DMM-AAE method effectively restrains the possible age with neuroimaging data can be used to detect accelerated atrophy after traumatic brain injury [2] and accelerated brain July26,2020.DateofpublicationAugust3,2020;dateofcurrentversionManuscriptreceivedMay22,2020;revisedJuly20,2020;accepted aging due to schizophrenia [3], type-2 diabetes mellitus [4], November 30, 2020. This work was supported in part by NIH under Grant and HIV disease [5]. Furthermore, prediction of brain age is MH116225,GrantMH117943,GrantMH104324,GrantMH109773,and also used to help discern possible environmental and lifestyle BabyConnectome ProjectConsortium.(Correspondingauthors:Grant1U01MH110274and inpart bytheEfforts of the UNC/UMN related influences on the human brain, e.g., younger brain Gang Li; Dinggang Shen.) age due to higher education, more self-reported physical Dan Hu, Han Zhang, Zhengwang Wu, Fan Wang, Li Wang, activity [6], and long-term meditation practice [7], as well as Radiology,Universityof NorthCarolinaat ChapelHill,ChapelHill,J.KeithSmith,WeiliLin,andGangLi arewith theDepartmentof increased brain age associated with midlife [8]. NC 27599 USA (e-mail: danhu@ad.unc.edu; hanzhang@med.unc.edu; Since different modalities of neuroimages can provide com-zhengwang_wu@med.unc.edu; fan_wang@med.unc.edu; li_wang@ plementary information to each other, researchers started to gang_li@med.unc.edu).med.unc.edu;jksmith@med.unc.edu; weili_lin@med.unc.edu; combine multimodal imaging to predict brain age. For exam- Dinggang Shen is with the Department of Radiology, University of ple, to capture cognitive impairment, functional connectivity NorthCarolinaatChapelHill,ChapelHill,NC27599USA,andalsowith derived from rs-fMRI and cortical morphological features SouthKorea(e-mail:dgshen@med.unc.edu).theDepartmentofArtificialIntelligence,KoreaUniversity,Seoul02841, from sMRI were combined for brain age prediction [9]. This article has supplementary downloadable material available at In [10], T2-weighted, T1-weighted, diffusion-weighted, and https://ieeexplore.ieee.org,providedbytheauthors. fluid-attenuate diversion recovery (FLAIR) scans were com-onlineathttps://ieeexplore.ieee.org.Colorversionsofoneormoreofthefiguresinthisarticleareavailable bined for age prediction, which highlighted the importance of Digital Object Identifier 10.1109/TMI.2020.3013825 using multimodal biomarkers to study normal aging. Publisher Copyright: {\textcopyright} 1982-2012 IEEE.",

year = "2020",

month = dec,

doi = "10.1109/TMI.2020.3013825",

language = "English",

volume = "39",

pages = "4137--4149",

journal = "IEEE Transactions on Medical Imaging",

issn = "0278-0062",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "12",

}

TY - JOUR

T1 - Disentangled-Multimodal Adversarial Autoencoder

T2 - Application to Infant Age Prediction with Incomplete Multimodal Neuroimages

AU - Hu, Dan

AU - Zhang, Han

AU - Wu, Zhengwang

AU - Wang, Fan

AU - Wang, Li

AU - Smith, J. Keith

AU - Lin, Weili

AU - Li, Gang

AU - Shen, Dinggang

N1 - Funding Information: This work was supported in part by NIH under Grant MH116225, Grant MH117943, Grant MH104324, Grant MH109773, and Grant 1U01MH110274 and in part by the Efforts of the UNC/UMN Baby Connectome Project Consortium. Funding Information: specific codes to represent the shared and complementary N I. INTRODUCTION information among modalities, respectively. Then, cross- EUROIMAGING-BASED age prediction is important ratio lossare designedasregularizationsto ensure thereconstructionrequirementandcommon-specificdistance for brain development analysis and early detection of effectiveness and thoroughness of the disentanglement. neurodevelopmental disorders [1]. The discrepancy between By arranging relatively independent autoencoders to the “chronological age” and the predicted “brain age” can separate the modalities and employing disentanglement be considered as an index of deviation from the normative undercross-reconstructionrequirementtointegratethem, developmental or aging trajectory. For example, the predicted our DMM-AAE method effectively restrains the possible age with neuroimaging data can be used to detect accelerated atrophy after traumatic brain injury [2] and accelerated brain July26,2020.DateofpublicationAugust3,2020;dateofcurrentversionManuscriptreceivedMay22,2020;revisedJuly20,2020;accepted aging due to schizophrenia [3], type-2 diabetes mellitus [4], November 30, 2020. This work was supported in part by NIH under Grant and HIV disease [5]. Furthermore, prediction of brain age is MH116225,GrantMH117943,GrantMH104324,GrantMH109773,and also used to help discern possible environmental and lifestyle BabyConnectome ProjectConsortium.(Correspondingauthors:Grant1U01MH110274and inpart bytheEfforts of the UNC/UMN related influences on the human brain, e.g., younger brain Gang Li; Dinggang Shen.) age due to higher education, more self-reported physical Dan Hu, Han Zhang, Zhengwang Wu, Fan Wang, Li Wang, activity [6], and long-term meditation practice [7], as well as Radiology,Universityof NorthCarolinaat ChapelHill,ChapelHill,J.KeithSmith,WeiliLin,andGangLi arewith theDepartmentof increased brain age associated with midlife [8]. NC 27599 USA (e-mail: danhu@ad.unc.edu; hanzhang@med.unc.edu; Since different modalities of neuroimages can provide com-zhengwang_wu@med.unc.edu; fan_wang@med.unc.edu; li_wang@ plementary information to each other, researchers started to gang_li@med.unc.edu).med.unc.edu;jksmith@med.unc.edu; weili_lin@med.unc.edu; combine multimodal imaging to predict brain age. For exam- Dinggang Shen is with the Department of Radiology, University of ple, to capture cognitive impairment, functional connectivity NorthCarolinaatChapelHill,ChapelHill,NC27599USA,andalsowith derived from rs-fMRI and cortical morphological features SouthKorea(e-mail:dgshen@med.unc.edu).theDepartmentofArtificialIntelligence,KoreaUniversity,Seoul02841, from sMRI were combined for brain age prediction [9]. This article has supplementary downloadable material available at In [10], T2-weighted, T1-weighted, diffusion-weighted, and https://ieeexplore.ieee.org,providedbytheauthors. fluid-attenuate diversion recovery (FLAIR) scans were com-onlineathttps://ieeexplore.ieee.org.Colorversionsofoneormoreofthefiguresinthisarticleareavailable bined for age prediction, which highlighted the importance of Digital Object Identifier 10.1109/TMI.2020.3013825 using multimodal biomarkers to study normal aging. Publisher Copyright: © 1982-2012 IEEE.

PY - 2020/12

Y1 - 2020/12

N2 - Effective fusion of structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) data has the potential to boost the accuracy of infant age prediction thanks to the complementary information provided by different imaging modalities. However, functional connectivity measured by fMRI during infancy is largely immature and noisy compared to the morphological features from sMRI, thus making the sMRI and fMRI fusion for infant brain analysis extremely challenging. With the conventional multimodal fusion strategies, adding fMRI data for age prediction has a high risk of introducing more noises than useful features, which would lead to reduced accuracy than that merely using sMRI data. To address this issue, we develop a novel model termed as disentangled-multimodal adversarial autoencoder (DMM-AAE) for infant age prediction based on multimodal brain MRI. Specifically, we disentangle the latent variables of autoencoder into common and specific codes to represent the shared and complementary information among modalities, respectively. Then, cross-reconstruction requirement and common-specific distance ratio loss are designed as regularizations to ensure the effectiveness and thoroughness of the disentanglement. By arranging relatively independent autoencoders to separate the modalities and employing disentanglement under cross-reconstruction requirement to integrate them, our DMM-AAE method effectively restrains the possible interference cross modalities, while realizing effective information fusion. Taking advantage of the latent variable disentanglement, a new strategy is further proposed and embedded into DMM-AAE to address the issue of incompleteness of the multimodal neuroimages, which can also be used as an independent algorithm for missing modality imputation. By taking six types of cortical morphometric features from sMRI and brain functional connectivity from fMRI as predictors, the superiority of the proposed DMM-AAE is validated on infant age (35 to 848 days after birth) prediction using incomplete multimodal neuroimages. The mean absolute error of the prediction based on DMM-AAE reaches 37.6 days, outperforming state-of-the-art methods. Generally, our proposed DMM-AAE can serve as a promising model for prediction with multimodal data.

AB - Effective fusion of structural magnetic resonance imaging (sMRI) and functional magnetic resonance imaging (fMRI) data has the potential to boost the accuracy of infant age prediction thanks to the complementary information provided by different imaging modalities. However, functional connectivity measured by fMRI during infancy is largely immature and noisy compared to the morphological features from sMRI, thus making the sMRI and fMRI fusion for infant brain analysis extremely challenging. With the conventional multimodal fusion strategies, adding fMRI data for age prediction has a high risk of introducing more noises than useful features, which would lead to reduced accuracy than that merely using sMRI data. To address this issue, we develop a novel model termed as disentangled-multimodal adversarial autoencoder (DMM-AAE) for infant age prediction based on multimodal brain MRI. Specifically, we disentangle the latent variables of autoencoder into common and specific codes to represent the shared and complementary information among modalities, respectively. Then, cross-reconstruction requirement and common-specific distance ratio loss are designed as regularizations to ensure the effectiveness and thoroughness of the disentanglement. By arranging relatively independent autoencoders to separate the modalities and employing disentanglement under cross-reconstruction requirement to integrate them, our DMM-AAE method effectively restrains the possible interference cross modalities, while realizing effective information fusion. Taking advantage of the latent variable disentanglement, a new strategy is further proposed and embedded into DMM-AAE to address the issue of incompleteness of the multimodal neuroimages, which can also be used as an independent algorithm for missing modality imputation. By taking six types of cortical morphometric features from sMRI and brain functional connectivity from fMRI as predictors, the superiority of the proposed DMM-AAE is validated on infant age (35 to 848 days after birth) prediction using incomplete multimodal neuroimages. The mean absolute error of the prediction based on DMM-AAE reaches 37.6 days, outperforming state-of-the-art methods. Generally, our proposed DMM-AAE can serve as a promising model for prediction with multimodal data.

KW - Infant age prediction

KW - autoencoder

KW - magnetic resonance imaging

KW - multimodal machine learning

UR - http://www.scopus.com/inward/record.url?scp=85097003419&partnerID=8YFLogxK

U2 - 10.1109/TMI.2020.3013825

DO - 10.1109/TMI.2020.3013825

M3 - Article

C2 - 32746154

AN - SCOPUS:85097003419

SN - 0278-0062

VL - 39

SP - 4137

EP - 4149

JO - IEEE Transactions on Medical Imaging

JF - IEEE Transactions on Medical Imaging

IS - 12

M1 - 9154745

ER -

Disentangled-Multimodal Adversarial Autoencoder: Application to Infant Age Prediction with Incomplete Multimodal Neuroimages

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this