Detection of Pilot’s Drowsiness Based on Multimodal Convolutional Bidirectional LSTM Network

Baek Woon Yu, Ji Hoon Jeong, Dae Hyeok Lee, Seong Whan Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)


The drowsiness of pilot causes the various aviation accidents such as an aircraft crash, breaking away airline, and passenger safety. Therefore, detecting the pilot’s drowsiness is one of the critical issues to prevent huge aircraft accidents and to predict pilot’s mental states. Conventional studies have been investigated using physiological signals such as brain signals, electrodermal activity (EDA), electrocardiogram (ECG), respiration (RESP) for detecting pilot’s drowsiness. However, these studies have not sufficient performance to prevent sudden aviation accidents yet because it could detect the mental states after drowsiness occurred and only focus on whether drowsiness or not. To overcome the limitations, in this paper, we propose a multimodal convolutional bidirectional LSTM network (MCBLN) to detect drowsiness or not as well as drowsiness level using the fused physiological signals (electroencephalography (EEG), EDA, ECG, and RESP) for the pilot’s environment. We acquired the physiological signals for the pilot’s simulated aircraft environment across seven participants. The proposed MCBLN extracted the features considering the spatial-temporal correlation of between EEG signals and peripheral physiological measures (PPMs) (EDA, ECG, RESP) to detect the current pilot’s drowsiness level. Our proposed method achieved the grand-averaged 45.16% (±1.01) classification accuracy for 9-level of drowsiness. Also, we obtained 84.41% (±1.34) classification accuracy for whether the drowsiness or not across all participants. Hence, we have demonstrated the possibility of the not only drowsiness detection but also 9-level of drowsiness for the pilot’s aircraft environment.

Original languageEnglish
Title of host publicationPattern Recognition - 5th Asian Conference, ACPR 2019, Revised Selected Papers
EditorsShivakumara Palaiahnakote, Gabriella Sanniti di Baja, Liang Wang, Wei Qi Yan
Number of pages14
ISBN (Print)9783030412982
Publication statusPublished - 2020
Event5th Asian Conference on Pattern Recognition, ACPR 2019 - Auckland, New Zealand
Duration: 2019 Nov 262019 Nov 29

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12047 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference5th Asian Conference on Pattern Recognition, ACPR 2019
Country/TerritoryNew Zealand


  • Drowsiness level detection
  • Multimodal convolutional bidirectional LSTM
  • Multimodal fusion
  • Pilot’s mental state

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'Detection of Pilot’s Drowsiness Based on Multimodal Convolutional Bidirectional LSTM Network'. Together they form a unique fingerprint.

Cite this