Multimodal Emotion Recognition Fusion Analysis Adapting BERT with Heterogeneous Feature Unification

Sanghyun Lee, David K. Han, Hanseok Ko

Research output: Contribution to journalArticlepeer-review

31 Citations (Scopus)


Human communication includes rich emotional content, thus the development of multimodal emotion recognition plays an important role in communication between humans and computers. Because of the complex emotional characteristics of a speaker, emotional recognition remains a challenge, particularly in capturing emotional cues across a variety of modalities, such as speech, facial expressions, and language. Audio and visual cues are particularly vital for a human observer in understanding emotions. However, most previous work on emotion recognition has been based solely on linguistic information, which can overlook various forms of nonverbal information. In this paper, we present a new multimodal emotion recognition approach that improves the BERT model for emotion recognition by combining it with heterogeneous features based on language, audio, and visual modalities. Specifically, we improve the BERT model due to the heterogeneous features of the audio and visual modalities. We introduce the Self-Multi-Attention Fusion module, Multi-Attention fusion module, and Video Fusion module, which are attention based multimodal fusion mechanisms using the recently proposed transformer architecture. We explore the optimal ways to combine fine-grained representations of audio and visual features into a common embedding while combining a pre-trained BERT model with modalities for fine-tuning. In our experiment, we evaluate the commonly used CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets for multimodal sentiment analysis. Ablation analysis indicates that the audio and visual components make a significant contribution to the recognition results, suggesting that these modalities contain highly complementary information for sentiment analysis based on video input. Our method shows that we achieve state-of-the-art performance on the CMU-MOSI, CMU-MOSEI, and IEMOCAP dataset.

Original languageEnglish
Article number9466122
Pages (from-to)94557-94572
Number of pages16
JournalIEEE Access
Publication statusPublished - 2021

Bibliographical note

Funding Information:
This work was supported by the Technology Innovation Program (or Industrial Strategic Technology Development Program, Development of Robot’s Natural Language Recognition and Emotional Dialogue Technology) through the Ministry of Trade, Industry and Energy (MOTIE), South Korea, under Grant -10073162.

Publisher Copyright:
© 2013 IEEE.


  • BERT
  • Multimodal emotion recognition
  • attention based multimodal
  • heterogeneous features
  • transformer

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering


Dive into the research topics of 'Multimodal Emotion Recognition Fusion Analysis Adapting BERT with Heterogeneous Feature Unification'. Together they form a unique fingerprint.

Cite this