Classification of Motion Sickness Levels using Multimodal Biosignals in Real Driving Conditions

Ji Un Hwang, Ji Seon Bang, Seong Whan Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Motion sickness is an unpleasant physiological response to situations involving the perception of motion. Research on motion sickness focuses on its manifestation by analyzing biosignals to observe physiological changes coinciding with the perception of motion sickness. Meanwhile, multimodal data fusion has gained attention for its ability to reflect the multimodality of real-life tasks and enhance the robustness of machine learning models. In this study, we aimed to find a deep learning-based multimodal framework for integrative analysis of multiple biosignals with the highest performance in classifying the level of carsickness. To do so, we first generated a dataset consisting of five different types of biosignals collected under real driving conditions: electroencephalogram (EEG), electrocardiogram (ECG), respiration (RESP), photoplethysmogram (PPG), and galvanic skin response (GSR). Then, we compared six deep learning-based unimodal classification models which have shown competency in signal classification. Lastly, we compared four different fusion methods for multimodal classification frameworks using either all five biosignals or three signals, which include RESP, ECG, and PPG. As a result, we found out that the fusion method combining self-attention and the tensor fusion network outperformed other unimodal and multimodal models with categorical accuracy of 76.26 % regardless of the number of biosignals used.

Original languageEnglish
Title of host publication2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1304-1309
Number of pages6
ISBN (Electronic)9781665452588
DOIs
Publication statusPublished - 2022
Event2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Prague, Czech Republic
Duration: 2022 Oct 92022 Oct 12

Publication series

NameConference Proceedings - IEEE International Conference on Systems, Man and Cybernetics
Volume2022-October
ISSN (Print)1062-922X

Conference

Conference2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022
Country/TerritoryCzech Republic
CityPrague
Period22/10/922/10/12

Bibliographical note

Funding Information:
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. 2017-0-00451: Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning, No. 2019-0-00079: Artificial Intelligence Graduate School Program, Korea University, No. 2021-0-02068: Artificial Intelligence Innovation Hub, and No. 2021-0-00866: Development of BMI application technology based on multiple bio-signals for autonomous vehicle drivers).

Publisher Copyright:
© 2022 IEEE.

Keywords

  • carsickness
  • motion sickness
  • multimodality

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Control and Systems Engineering
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of 'Classification of Motion Sickness Levels using Multimodal Biosignals in Real Driving Conditions'. Together they form a unique fingerprint.

Cite this