Classification of Motion Sickness Levels using Multimodal Biosignals in Real Driving Conditions

Ji Un Hwang, Ji Seon Bang, Seong Whan Lee

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    9 Citations (Scopus)

    Abstract

    Motion sickness is an unpleasant physiological response to situations involving the perception of motion. Research on motion sickness focuses on its manifestation by analyzing biosignals to observe physiological changes coinciding with the perception of motion sickness. Meanwhile, multimodal data fusion has gained attention for its ability to reflect the multimodality of real-life tasks and enhance the robustness of machine learning models. In this study, we aimed to find a deep learning-based multimodal framework for integrative analysis of multiple biosignals with the highest performance in classifying the level of carsickness. To do so, we first generated a dataset consisting of five different types of biosignals collected under real driving conditions: electroencephalogram (EEG), electrocardiogram (ECG), respiration (RESP), photoplethysmogram (PPG), and galvanic skin response (GSR). Then, we compared six deep learning-based unimodal classification models which have shown competency in signal classification. Lastly, we compared four different fusion methods for multimodal classification frameworks using either all five biosignals or three signals, which include RESP, ECG, and PPG. As a result, we found out that the fusion method combining self-attention and the tensor fusion network outperformed other unimodal and multimodal models with categorical accuracy of 76.26 % regardless of the number of biosignals used.

    Original languageEnglish
    Title of host publication2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Proceedings
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Pages1304-1309
    Number of pages6
    ISBN (Electronic)9781665452588
    DOIs
    Publication statusPublished - 2022
    Event2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Prague, Czech Republic
    Duration: 2022 Oct 92022 Oct 12

    Publication series

    NameConference Proceedings - IEEE International Conference on Systems, Man and Cybernetics
    Volume2022-October
    ISSN (Print)1062-922X

    Conference

    Conference2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022
    Country/TerritoryCzech Republic
    CityPrague
    Period22/10/922/10/12

    Bibliographical note

    Funding Information:
    This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. 2017-0-00451: Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning, No. 2019-0-00079: Artificial Intelligence Graduate School Program, Korea University, No. 2021-0-02068: Artificial Intelligence Innovation Hub, and No. 2021-0-00866: Development of BMI application technology based on multiple bio-signals for autonomous vehicle drivers).

    Publisher Copyright:
    © 2022 IEEE.

    Keywords

    • carsickness
    • motion sickness
    • multimodality

    ASJC Scopus subject areas

    • Electrical and Electronic Engineering
    • Control and Systems Engineering
    • Human-Computer Interaction

    Fingerprint

    Dive into the research topics of 'Classification of Motion Sickness Levels using Multimodal Biosignals in Real Driving Conditions'. Together they form a unique fingerprint.

    Cite this