Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals

Ji Hoon Jeong; Kyung Hwan Shim; Dong Joo Kim; Seong Whan Lee

doi:10.1109/TNSRE.2020.2981659

Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals

Ji Hoon Jeong, Kyung Hwan Shim, Dong Joo Kim, Seong Whan Lee

Research output: Contribution to journal › Article › peer-review

166 Citations (Scopus)

Abstract

Brain-machine interfaces (BMIs) can be used to decode brain activity into commands to control external devices. This paper presents the decoding of intuitive upper extremity imagery for multi-directional arm reaching tasks in three-dimensional (3D) environments. We designed and implemented an experimental environment in which electroencephalogram (EEG) signals can be acquired for movement execution and imagery. Fifteen subjects participated in our experiments. We proposed a multi-directional convolution neural network-bidirectional long short-term memory network (MDCBN)-based deep learning framework. The decoding performances for six directions in 3D space were measured by the correlation coefficient (CC) and the normalized root mean square error (NRMSE) between predicted and baseline velocity profiles. The grand-averaged CCs of multi-direction were 0.47 and 0.45 for the execution and imagery sessions, respectively, across all subjects. The NRMSE values were below 0.2 for both sessions. Furthermore, in this study, the proposed MDCBN was evaluated by two online experiments for real-time robotic arm control, and the grand-averaged success rates were approximately 0.60 (±0.14) and 0.43 (±0.09), respectively. Hence, we demonstrate the feasibility of intuitive robotic arm control based on EEG signals for real-world environments.

Original language	English
Article number	9040397
Pages (from-to)	1226-1238
Number of pages	13
Journal	IEEE Transactions on Neural Systems and Rehabilitation Engineering
Volume	28
Issue number	5
DOIs	https://doi.org/10.1109/TNSRE.2020.2981659
Publication status	Published - 2020 May

Bibliographical note

Funding Information:
Manuscript received July 11, 2019; revised November 19, 2019 and February 19, 2020; accepted March 12, 2020. Date of publication March 18, 2020; date of current version May 8, 2020. This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) funded by the Korea Government through the Development of Non-Invasive Integrated BCI SW Platform to Control Home Appliances and External Devices by User’s Thought via AR/VR Interface under Grant 2017-0-00432, in part by the IITP funded by the Korea Government through the Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning under Grant 2017-0-00451, and in part by the IITP funded by the Korea Government through the Department of Artificial Intelligence, Korea University, under Grant 2019-0-00079. (Corresponding author: Seong-Whan Lee.) Ji-Hoon Jeong, Kyung-Hwan Shim, and Dong-Joo Kim are with the Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, South Korea (e-mail: jh_jeong@korea.ac.kr; kh_shim@korea.ac.kr; dongjookim@korea.ac.kr).

Funding Information:
This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) funded by the Korea Government through the Development of Non-Invasive Integrated BCI SW Platform to Control Home Appliances and External Devices by User's Thought via AR/VR Interface under Grant 2017-0-00432, in part by the IITP funded by the Korea Government through the Development of BCI based Brain and Cognitive Computing Technology for Recognizing User's Intentions using Deep Learning under Grant 2017-0-00451, and in part by the IITP funded by the Korea Government through the Department of Artificial Intelligence, Korea University, under Grant 2019-0-00079.

Publisher Copyright:
© 2001-2011 IEEE.

Keywords

Brain-machine interface (BMI)
deep learning
electroencephalogram (EEG)
intuitive robotic arm control
motor imagery

ASJC Scopus subject areas

Internal Medicine
Neuroscience(all)
Biomedical Engineering
Rehabilitation

Access to Document

10.1109/TNSRE.2020.2981659

Cite this

@article{071175a5577f49dc8b0f901b1a122352,

title = "Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals",

abstract = "Brain-machine interfaces (BMIs) can be used to decode brain activity into commands to control external devices. This paper presents the decoding of intuitive upper extremity imagery for multi-directional arm reaching tasks in three-dimensional (3D) environments. We designed and implemented an experimental environment in which electroencephalogram (EEG) signals can be acquired for movement execution and imagery. Fifteen subjects participated in our experiments. We proposed a multi-directional convolution neural network-bidirectional long short-term memory network (MDCBN)-based deep learning framework. The decoding performances for six directions in 3D space were measured by the correlation coefficient (CC) and the normalized root mean square error (NRMSE) between predicted and baseline velocity profiles. The grand-averaged CCs of multi-direction were 0.47 and 0.45 for the execution and imagery sessions, respectively, across all subjects. The NRMSE values were below 0.2 for both sessions. Furthermore, in this study, the proposed MDCBN was evaluated by two online experiments for real-time robotic arm control, and the grand-averaged success rates were approximately 0.60 (±0.14) and 0.43 (±0.09), respectively. Hence, we demonstrate the feasibility of intuitive robotic arm control based on EEG signals for real-world environments.",

keywords = "Brain-machine interface (BMI), deep learning, electroencephalogram (EEG), intuitive robotic arm control, motor imagery",

author = "Jeong, {Ji Hoon} and Shim, {Kyung Hwan} and Kim, {Dong Joo} and Lee, {Seong Whan}",

note = "Funding Information: Manuscript received July 11, 2019; revised November 19, 2019 and February 19, 2020; accepted March 12, 2020. Date of publication March 18, 2020; date of current version May 8, 2020. This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) funded by the Korea Government through the Development of Non-Invasive Integrated BCI SW Platform to Control Home Appliances and External Devices by User{\textquoteright}s Thought via AR/VR Interface under Grant 2017-0-00432, in part by the IITP funded by the Korea Government through the Development of BCI based Brain and Cognitive Computing Technology for Recognizing User{\textquoteright}s Intentions using Deep Learning under Grant 2017-0-00451, and in part by the IITP funded by the Korea Government through the Department of Artificial Intelligence, Korea University, under Grant 2019-0-00079. (Corresponding author: Seong-Whan Lee.) Ji-Hoon Jeong, Kyung-Hwan Shim, and Dong-Joo Kim are with the Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, South Korea (e-mail: jh_jeong@korea.ac.kr; kh_shim@korea.ac.kr; dongjookim@korea.ac.kr). Funding Information: This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) funded by the Korea Government through the Development of Non-Invasive Integrated BCI SW Platform to Control Home Appliances and External Devices by User's Thought via AR/VR Interface under Grant 2017-0-00432, in part by the IITP funded by the Korea Government through the Development of BCI based Brain and Cognitive Computing Technology for Recognizing User's Intentions using Deep Learning under Grant 2017-0-00451, and in part by the IITP funded by the Korea Government through the Department of Artificial Intelligence, Korea University, under Grant 2019-0-00079. Publisher Copyright: {\textcopyright} 2001-2011 IEEE.",

year = "2020",

month = may,

doi = "10.1109/TNSRE.2020.2981659",

language = "English",

volume = "28",

pages = "1226--1238",

journal = "IEEE Transactions on Neural Systems and Rehabilitation Engineering",

issn = "1534-4320",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "5",

}

TY - JOUR

T1 - Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals

AU - Jeong, Ji Hoon

AU - Shim, Kyung Hwan

AU - Kim, Dong Joo

AU - Lee, Seong Whan

N1 - Funding Information: Manuscript received July 11, 2019; revised November 19, 2019 and February 19, 2020; accepted March 12, 2020. Date of publication March 18, 2020; date of current version May 8, 2020. This work was supported in part by the Institute of Information & Communications Technology Planning & Evaluation (IITP) funded by the Korea Government through the Development of Non-Invasive Integrated BCI SW Platform to Control Home Appliances and External Devices by User’s Thought via AR/VR Interface under Grant 2017-0-00432, in part by the IITP funded by the Korea Government through the Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning under Grant 2017-0-00451, and in part by the IITP funded by the Korea Government through the Department of Artificial Intelligence, Korea University, under Grant 2019-0-00079. (Corresponding author: Seong-Whan Lee.) Ji-Hoon Jeong, Kyung-Hwan Shim, and Dong-Joo Kim are with the Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, South Korea (e-mail: jh_jeong@korea.ac.kr; kh_shim@korea.ac.kr; dongjookim@korea.ac.kr). Funding Information: This work was supported in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP) funded by the Korea Government through the Development of Non-Invasive Integrated BCI SW Platform to Control Home Appliances and External Devices by User's Thought via AR/VR Interface under Grant 2017-0-00432, in part by the IITP funded by the Korea Government through the Development of BCI based Brain and Cognitive Computing Technology for Recognizing User's Intentions using Deep Learning under Grant 2017-0-00451, and in part by the IITP funded by the Korea Government through the Department of Artificial Intelligence, Korea University, under Grant 2019-0-00079. Publisher Copyright: © 2001-2011 IEEE.

PY - 2020/5

Y1 - 2020/5

N2 - Brain-machine interfaces (BMIs) can be used to decode brain activity into commands to control external devices. This paper presents the decoding of intuitive upper extremity imagery for multi-directional arm reaching tasks in three-dimensional (3D) environments. We designed and implemented an experimental environment in which electroencephalogram (EEG) signals can be acquired for movement execution and imagery. Fifteen subjects participated in our experiments. We proposed a multi-directional convolution neural network-bidirectional long short-term memory network (MDCBN)-based deep learning framework. The decoding performances for six directions in 3D space were measured by the correlation coefficient (CC) and the normalized root mean square error (NRMSE) between predicted and baseline velocity profiles. The grand-averaged CCs of multi-direction were 0.47 and 0.45 for the execution and imagery sessions, respectively, across all subjects. The NRMSE values were below 0.2 for both sessions. Furthermore, in this study, the proposed MDCBN was evaluated by two online experiments for real-time robotic arm control, and the grand-averaged success rates were approximately 0.60 (±0.14) and 0.43 (±0.09), respectively. Hence, we demonstrate the feasibility of intuitive robotic arm control based on EEG signals for real-world environments.

AB - Brain-machine interfaces (BMIs) can be used to decode brain activity into commands to control external devices. This paper presents the decoding of intuitive upper extremity imagery for multi-directional arm reaching tasks in three-dimensional (3D) environments. We designed and implemented an experimental environment in which electroencephalogram (EEG) signals can be acquired for movement execution and imagery. Fifteen subjects participated in our experiments. We proposed a multi-directional convolution neural network-bidirectional long short-term memory network (MDCBN)-based deep learning framework. The decoding performances for six directions in 3D space were measured by the correlation coefficient (CC) and the normalized root mean square error (NRMSE) between predicted and baseline velocity profiles. The grand-averaged CCs of multi-direction were 0.47 and 0.45 for the execution and imagery sessions, respectively, across all subjects. The NRMSE values were below 0.2 for both sessions. Furthermore, in this study, the proposed MDCBN was evaluated by two online experiments for real-time robotic arm control, and the grand-averaged success rates were approximately 0.60 (±0.14) and 0.43 (±0.09), respectively. Hence, we demonstrate the feasibility of intuitive robotic arm control based on EEG signals for real-world environments.

KW - Brain-machine interface (BMI)

KW - deep learning

KW - electroencephalogram (EEG)

KW - intuitive robotic arm control

KW - motor imagery

UR - http://www.scopus.com/inward/record.url?scp=85084003251&partnerID=8YFLogxK

U2 - 10.1109/TNSRE.2020.2981659

DO - 10.1109/TNSRE.2020.2981659

M3 - Article

C2 - 32191894

AN - SCOPUS:85084003251

SN - 1534-4320

VL - 28

SP - 1226

EP - 1238

JO - IEEE Transactions on Neural Systems and Rehabilitation Engineering

JF - IEEE Transactions on Neural Systems and Rehabilitation Engineering

IS - 5

M1 - 9040397

ER -

Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this