Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation

Ji Seon Bang; Seong Whan Lee

doi:10.1007/978-3-031-02375-0_8

Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation

Ji Seon Bang, Seong Whan Lee

Department of Artificial Intelligence

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Recently, various deep neural networks have been applied to classify electroencephalogram (EEG) signal. EEG is a brain signal that can be acquired in a non-invasive way and has a high temporal resolution. It can be used to decode the intention of users. As the EEG signal has a high dimension of feature space, appropriate feature extraction methods are needed to improve classification performance. In this study, we obtained spatio-temporal feature representation and classified them with the combined convolutional neural networks (CNN)-gated recurrent unit (GRU) model. To this end, we obtained covariance matrices in each different temporal band and then concatenated them on the temporal axis to obtain a final spatio-temporal feature representation. In the classification model, CNN is responsible for spatial feature extraction and GRU is responsible for temporal feature extraction. Classification performance was improved by distinguishing spatial data processing and temporal data processing. The average accuracy of the proposed model was 77.70% (±15.39) for the BCI competition IV_2a data set. The proposed method outperformed all other methods compared as a baseline method.

Original language	English
Title of host publication	Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers
Editors	Christian Wallraven, Qingshan Liu, Hajime Nagahara
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	104-115
Number of pages	12
ISBN (Print)	9783031023743
DOIs	https://doi.org/10.1007/978-3-031-02375-0_8
Publication status	Published - 2022
Event	6th Asian Conference on Pattern Recognition, ACPR 2021 - Virtual, Online Duration: 2021 Nov 9 → 2021 Nov 12

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13188 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	6th Asian Conference on Pattern Recognition, ACPR 2021
City	Virtual, Online
Period	21/11/9 → 21/11/12

Bibliographical note

Funding Information:
Keywords: Brain-computer interface (BCI) · Electroencephalography (EEG) · Motor imagery (MI) · Convolutional neural network (CNN) · Gated recurrent unit (GRU) This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2015-0-00185, Development of Intelligent Pattern Recognition Softwares for Ambulatory Brain Computer Interface, No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning, No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University)).

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

Keywords

Brain-computer interface (BCI)
Convolutional neural network (CNN)
Electroencephalography (EEG)
Gated recurrent unit (GRU)
Motor imagery (MI)

ASJC Scopus subject areas

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-031-02375-0_8

Cite this

Bang, J. S., & Lee, S. W. (2022). Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation. In C. Wallraven, Q. Liu, & H. Nagahara (Eds.), Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers (pp. 104-115). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13188 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-02375-0_8

Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation. / Bang, Ji Seon; Lee, Seong Whan.
Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. ed. / Christian Wallraven; Qingshan Liu; Hajime Nagahara. Springer Science and Business Media Deutschland GmbH, 2022. p. 104-115 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13188 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Bang, JS & Lee, SW 2022, Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation. in C Wallraven, Q Liu & H Nagahara (eds), Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13188 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 104-115, 6th Asian Conference on Pattern Recognition, ACPR 2021, Virtual, Online, 21/11/9. https://doi.org/10.1007/978-3-031-02375-0_8

Bang JS, Lee SW. Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation. In Wallraven C, Liu Q, Nagahara H, editors, Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2022. p. 104-115. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-02375-0_8

Bang, Ji Seon ; Lee, Seong Whan. / Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation. Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers. editor / Christian Wallraven ; Qingshan Liu ; Hajime Nagahara. Springer Science and Business Media Deutschland GmbH, 2022. pp. 104-115 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{fe06f412a3f14033b7f0b5b6ed5e7551,

title = "Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation",

abstract = "Recently, various deep neural networks have been applied to classify electroencephalogram (EEG) signal. EEG is a brain signal that can be acquired in a non-invasive way and has a high temporal resolution. It can be used to decode the intention of users. As the EEG signal has a high dimension of feature space, appropriate feature extraction methods are needed to improve classification performance. In this study, we obtained spatio-temporal feature representation and classified them with the combined convolutional neural networks (CNN)-gated recurrent unit (GRU) model. To this end, we obtained covariance matrices in each different temporal band and then concatenated them on the temporal axis to obtain a final spatio-temporal feature representation. In the classification model, CNN is responsible for spatial feature extraction and GRU is responsible for temporal feature extraction. Classification performance was improved by distinguishing spatial data processing and temporal data processing. The average accuracy of the proposed model was 77.70% (±15.39) for the BCI competition IV_2a data set. The proposed method outperformed all other methods compared as a baseline method.",

keywords = "Brain-computer interface (BCI), Convolutional neural network (CNN), Electroencephalography (EEG), Gated recurrent unit (GRU), Motor imagery (MI)",

author = "Bang, {Ji Seon} and Lee, {Seong Whan}",

note = "Funding Information: Keywords: Brain-computer interface (BCI) · Electroencephalography (EEG) · Motor imagery (MI) · Convolutional neural network (CNN) · Gated recurrent unit (GRU) This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2015-0-00185, Development of Intelligent Pattern Recognition Softwares for Ambulatory Brain Computer Interface, No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User{\textquoteright}s Intentions using Deep Learning, No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University)). Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.; 6th Asian Conference on Pattern Recognition, ACPR 2021 ; Conference date: 09-11-2021 Through 12-11-2021",

year = "2022",

doi = "10.1007/978-3-031-02375-0_8",

language = "English",

isbn = "9783031023743",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "104--115",

editor = "Christian Wallraven and Qingshan Liu and Hajime Nagahara",

booktitle = "Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers",

address = "Germany",

}

TY - GEN

T1 - Motor Imagery Classification Based on CNN-GRU Network with Spatio-Temporal Feature Representation

AU - Bang, Ji Seon

AU - Lee, Seong Whan

N1 - Funding Information: Keywords: Brain-computer interface (BCI) · Electroencephalography (EEG) · Motor imagery (MI) · Convolutional neural network (CNN) · Gated recurrent unit (GRU) This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2015-0-00185, Development of Intelligent Pattern Recognition Softwares for Ambulatory Brain Computer Interface, No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning, No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University)). Publisher Copyright: © 2022, Springer Nature Switzerland AG.

PY - 2022

Y1 - 2022

N2 - Recently, various deep neural networks have been applied to classify electroencephalogram (EEG) signal. EEG is a brain signal that can be acquired in a non-invasive way and has a high temporal resolution. It can be used to decode the intention of users. As the EEG signal has a high dimension of feature space, appropriate feature extraction methods are needed to improve classification performance. In this study, we obtained spatio-temporal feature representation and classified them with the combined convolutional neural networks (CNN)-gated recurrent unit (GRU) model. To this end, we obtained covariance matrices in each different temporal band and then concatenated them on the temporal axis to obtain a final spatio-temporal feature representation. In the classification model, CNN is responsible for spatial feature extraction and GRU is responsible for temporal feature extraction. Classification performance was improved by distinguishing spatial data processing and temporal data processing. The average accuracy of the proposed model was 77.70% (±15.39) for the BCI competition IV_2a data set. The proposed method outperformed all other methods compared as a baseline method.

AB - Recently, various deep neural networks have been applied to classify electroencephalogram (EEG) signal. EEG is a brain signal that can be acquired in a non-invasive way and has a high temporal resolution. It can be used to decode the intention of users. As the EEG signal has a high dimension of feature space, appropriate feature extraction methods are needed to improve classification performance. In this study, we obtained spatio-temporal feature representation and classified them with the combined convolutional neural networks (CNN)-gated recurrent unit (GRU) model. To this end, we obtained covariance matrices in each different temporal band and then concatenated them on the temporal axis to obtain a final spatio-temporal feature representation. In the classification model, CNN is responsible for spatial feature extraction and GRU is responsible for temporal feature extraction. Classification performance was improved by distinguishing spatial data processing and temporal data processing. The average accuracy of the proposed model was 77.70% (±15.39) for the BCI competition IV_2a data set. The proposed method outperformed all other methods compared as a baseline method.

KW - Brain-computer interface (BCI)

KW - Convolutional neural network (CNN)

KW - Electroencephalography (EEG)

KW - Gated recurrent unit (GRU)

KW - Motor imagery (MI)

UR - http://www.scopus.com/inward/record.url?scp=85130395433&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-02375-0_8

DO - 10.1007/978-3-031-02375-0_8

M3 - Conference contribution

AN - SCOPUS:85130395433

SN - 9783031023743

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 104

EP - 115

BT - Pattern Recognition - 6th Asian Conference, ACPR 2021, Revised Selected Papers

A2 - Wallraven, Christian

A2 - Liu, Qingshan

A2 - Nagahara, Hajime

PB - Springer Science and Business Media Deutschland GmbH

T2 - 6th Asian Conference on Pattern Recognition, ACPR 2021

Y2 - 9 November 2021 through 12 November 2021

ER -