Multimodal Emotion Recognition Fusion Analysis Adapting BERT with Heterogeneous Feature Unification

Sanghyun Lee, David K. Han, Hanseok Ko

    Research output: Contribution to journalArticlepeer-review

    54 Citations (Scopus)

    Abstract

    Human communication includes rich emotional content, thus the development of multimodal emotion recognition plays an important role in communication between humans and computers. Because of the complex emotional characteristics of a speaker, emotional recognition remains a challenge, particularly in capturing emotional cues across a variety of modalities, such as speech, facial expressions, and language. Audio and visual cues are particularly vital for a human observer in understanding emotions. However, most previous work on emotion recognition has been based solely on linguistic information, which can overlook various forms of nonverbal information. In this paper, we present a new multimodal emotion recognition approach that improves the BERT model for emotion recognition by combining it with heterogeneous features based on language, audio, and visual modalities. Specifically, we improve the BERT model due to the heterogeneous features of the audio and visual modalities. We introduce the Self-Multi-Attention Fusion module, Multi-Attention fusion module, and Video Fusion module, which are attention based multimodal fusion mechanisms using the recently proposed transformer architecture. We explore the optimal ways to combine fine-grained representations of audio and visual features into a common embedding while combining a pre-trained BERT model with modalities for fine-tuning. In our experiment, we evaluate the commonly used CMU-MOSI, CMU-MOSEI, and IEMOCAP datasets for multimodal sentiment analysis. Ablation analysis indicates that the audio and visual components make a significant contribution to the recognition results, suggesting that these modalities contain highly complementary information for sentiment analysis based on video input. Our method shows that we achieve state-of-the-art performance on the CMU-MOSI, CMU-MOSEI, and IEMOCAP dataset.

    Original languageEnglish
    Article number9466122
    Pages (from-to)94557-94572
    Number of pages16
    JournalIEEE Access
    Volume9
    DOIs
    Publication statusPublished - 2021

    Bibliographical note

    Publisher Copyright:
    © 2013 IEEE.

    Keywords

    • BERT
    • Multimodal emotion recognition
    • attention based multimodal
    • heterogeneous features
    • transformer

    ASJC Scopus subject areas

    • General Computer Science
    • General Materials Science
    • General Engineering

    Fingerprint

    Dive into the research topics of 'Multimodal Emotion Recognition Fusion Analysis Adapting BERT with Heterogeneous Feature Unification'. Together they form a unique fingerprint.

    Cite this