Motion sickness is an unpleasant physiological response to situations involving the perception of motion. Research on motion sickness focuses on its manifestation by analyzing biosignals to observe physiological changes coinciding with the perception of motion sickness. Meanwhile, multimodal data fusion has gained attention for its ability to reflect the multimodality of real-life tasks and enhance the robustness of machine learning models. In this study, we aimed to find a deep learning-based multimodal framework for integrative analysis of multiple biosignals with the highest performance in classifying the level of carsickness. To do so, we first generated a dataset consisting of five different types of biosignals collected under real driving conditions: electroencephalogram (EEG), electrocardiogram (ECG), respiration (RESP), photoplethysmogram (PPG), and galvanic skin response (GSR). Then, we compared six deep learning-based unimodal classification models which have shown competency in signal classification. Lastly, we compared four different fusion methods for multimodal classification frameworks using either all five biosignals or three signals, which include RESP, ECG, and PPG. As a result, we found out that the fusion method combining self-attention and the tensor fusion network outperformed other unimodal and multimodal models with categorical accuracy of 76.26 % regardless of the number of biosignals used.
|Title of host publication||2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Proceedings|
|Publisher||Institute of Electrical and Electronics Engineers Inc.|
|Number of pages||6|
|Publication status||Published - 2022|
|Event||2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022 - Prague, Czech Republic|
Duration: 2022 Oct 9 → 2022 Oct 12
|Name||Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics|
|Conference||2022 IEEE International Conference on Systems, Man, and Cybernetics, SMC 2022|
|Period||22/10/9 → 22/10/12|
Bibliographical noteFunding Information:
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grants funded by the Korea government (MSIT) (No. 2017-0-00451: Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning, No. 2019-0-00079: Artificial Intelligence Graduate School Program, Korea University, No. 2021-0-02068: Artificial Intelligence Innovation Hub, and No. 2021-0-00866: Development of BMI application technology based on multiple bio-signals for autonomous vehicle drivers).
© 2022 IEEE.
- motion sickness
ASJC Scopus subject areas
- Electrical and Electronic Engineering
- Control and Systems Engineering
- Human-Computer Interaction