Abstract
Motion sickness is an unpleasant physiological response to situations involving the perception of motion. Research on motion sickness focuses on its manifestation by analyzing biosignals to observe physiological changes coinciding with the perception of motion sickness. Meanwhile, multimodal data fusion has gained attention for its ability to reflect the multimodality of real-life tasks and enhance the robustness of machine learning models. In this study, we aimed to find a deep learning-based multimodal framework for integrative analysis of multiple biosignals with the highest performance in classifying the level of carsickness. To do so, we first generated a dataset consisting of five different types of biosignals collected under real driving conditions: electroencephalogram (EEG), electrocardiogram (ECG), respiration (RESP), photoplethysmogram (PPG), and galvanic skin response (GSR). Then, we compared six deep learning-based unimodal classification models which have shown competency in signal classification. Lastly, we compared four different fusion methods for multimodal classification frameworks using either all five biosignals or three signals, which include RESP, ECG, and PPG. As a result, we found out that the fusion method combining self-attention and the tensor fusion network outperformed other unimodal and multimodal models with categorical accuracy of 76.26 % regardless of the number of biosignals used.
Original language | English |
---|---|
Title of host publication | 2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Proceedings |
Publisher | Institute of Electrical and Electronics Engineers Inc. |
Pages | 1304-1309 |
Number of pages | 6 |
ISBN (Electronic) | 9781665452588 |
DOIs | |
Publication status | Published - 2022 |
Event | 2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Prague, Czech Republic Duration: 2022 Oct 9 → 2022 Oct 12 |
Publication series
Name | Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics |
---|---|
Volume | 2022-October |
ISSN (Print) | 1062-922X |
Conference
Conference | 2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 |
---|---|
Country/Territory | Czech Republic |
City | Prague |
Period | 22/10/9 → 22/10/12 |
Bibliographical note
Funding Information:This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. 2017-0-00451: Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning, No. 2019-0-00079: Artificial Intelligence Graduate School Program, Korea University, No. 2021-0-02068: Artificial Intelligence Innovation Hub, and No. 2021-0-00866: Development of BMI application technology based on multiple bio-signals for autonomous vehicle drivers).
Publisher Copyright:
© 2022 IEEE.
Keywords
- carsickness
- motion sickness
- multimodality
ASJC Scopus subject areas
- Electrical and Electronic Engineering
- Control and Systems Engineering
- Human-Computer Interaction