Detection of Pilot’s Drowsiness Based on Multimodal Convolutional Bidirectional LSTM Network

The drowsiness of pilot causes the various aviation accidents such as an aircraft crash, breaking away airline, and passenger safety. Therefore, detecting the pilot’s drowsiness is one of the critical issues to prevent huge aircraft accidents and to predict pilot’s mental states. Conventional studies have been investigated using physiological signals such as brain signals, electrodermal activity (EDA), electrocardiogram (ECG), respiration (RESP) for detecting pilot’s drowsiness. However, these studies have not sufficient performance to prevent sudden aviation accidents yet because it could detect the mental states after drowsiness occurred and only focus on whether drowsiness or not. To overcome the limitations, in this paper, we propose a multimodal convolutional bidirectional LSTM network (MCBLN) to detect drowsiness or not as well as drowsiness level using the fused physiological signals (electroencephalography (EEG), EDA, ECG, and RESP) for the pilot’s environment. We acquired the physiological signals for the pilot’s simulated aircraft environment across seven participants. The proposed MCBLN extracted the features considering the spatial-temporal correlation of between EEG signals and peripheral physiological measures (PPMs) (EDA, ECG, RESP) to detect the current pilot’s drowsiness level. Our proposed method achieved the grand-averaged 45.16% (±1.01) classification accuracy for 9-level of drowsiness. Also, we obtained 84.41% (±1.34) classification accuracy for whether the drowsiness or not across all participants. Hence, we have demonstrated the possibility of the not only drowsiness detection but also 9-level of drowsiness for the pilot’s aircraft environment.