Multilevel Feature Fusion with 3D Convolutional Neural Network for EEG-Based Workload Estimation

Youngchul Kwak, Kyeongbo Kong, Woo Jin Song, Byoung Kyong Min, Seong Eun Kim

Research output: Contribution to journalArticlepeer-review

28 Citations (Scopus)


Mental workload is defined as the proportion of the information processing capability used to perform a task. High cognitive load requires additional resources to process information; this demand for additional resources may reduce the processing efficiency and performance. Therefore, the technique of workload estimation can ensure a proper working environment to promote the working efficiency of each person. In this paper, we propose a three-dimensional convolutional neural network (3D CNN) employing a multilevel feature fusion algorithm for mental workload estimation using electroencephalogram (EEG) signals. The 1D EEG signals are converted to 3D EEG images to enable the 3D CNN to learn the spectral and spatial information over the scalp. The multilevel feature fusion framework integrates local and global neuronal activities by workload tasks in the 3D CNN algorithm. Multilevel features are extracted in each layer of the 3D convolution operation and each multilevel feature is multiplied by a weighting factor, which determines the importance of the feature. The weighting factor is adaptively estimated for each EEG image by a backpropagation process. Furthermore, we generate subframes from each EEG image and propose a temporal attention technique based on the long short-term memory model (LSTM) to extract a significant subframe at each multilevel feature that is strongly correlated with task difficulty. To verify the performance of our network, we performed the Sternberg task to measure the mental workload of the participant, which was classified according to its difficulty as low or high workload condition. We showed that the difficulty of the workload was well designed, which was reflected in the behavior of the participant. Our network is trained on this dataset and the accuracy of our network is 90.8 %, which is better than that of conventional algorithms. We also evaluated our method using the public EEG dataset and achieved 93.9 % accuracy.

Original languageEnglish
Article number8960298
Pages (from-to)16009-16021
Number of pages13
JournalIEEE Access
Publication statusPublished - 2020

Bibliographical note

Funding Information:
This work was supported in part by the National Research Foundation of Korea (NRF) Grant funded by Korea Government (MSIT), under Grant 2017R1C1B5017254, Grant 2018M3C1B8017549, and Grant 2018R1A2B6004084, and in part by the Institute for Information & Communications Technology Promotion (IITP) through the Korea Government under Grant 2017-0-00451, Grant 2017-0-00432, and Grant 2019-2016-0-00464.

Publisher Copyright:
© 2013 IEEE.


  • Convolutional neural network
  • electroencephalogram (EEG)
  • feature fusion
  • mental workload
  • working memory

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering


Dive into the research topics of 'Multilevel Feature Fusion with 3D Convolutional Neural Network for EEG-Based Workload Estimation'. Together they form a unique fingerprint.

Cite this