Epoch-level and Sequence-level Multi-Head Self-Attention-based Sleep Stage Classification

Koohong Jung; Moogyeong Kim; Wonzoo Chung

doi:10.1109/BCI57258.2023.10078700

Epoch-level and Sequence-level Multi-Head Self-Attention-based Sleep Stage Classification

Koohong Jung, Moogyeong Kim, Wonzoo Chung

Department of Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

Inthis paper, we propose a Transformer-based sleep stage classification method by effectively combining epoch-level and sequence-level multi-head self-attention with multiple outputs to aid the training. Most of the existing automated sleep staging methods utilized convolutional neural networks (CNNs) for feature extraction and recurrent neural networks (RNNs) or Transformer for capturing global dependencies. However, there are disadvantages when utilizing CNNs and RNNs (e.g., large parameter sizes, information loss). Therefore, Transformer-only architecture with epoch-level multi-head self-attention for capturing intra-epoch dependencies and sequence-level multihead self-attention for capturing inter-epoch dependencies in 30-second PSG signals is utilized. Furthermore, a sequence of learnable class tokens is deployed to store summary of each sleep epoch. As a result, three different outputs are produced, in which two are used as references during the backpropagation in training. Numerical simulation results on the Sleep-EDFX78 dataset confirm that the proposed method show improvements compared to the existing Transformer-based methods.

Original language	English
Title of host publication	11th International Winter Conference on Brain-Computer Interface, BCI 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665464444
DOIs	https://doi.org/10.1109/BCI57258.2023.10078700
Publication status	Published - 2023
Event	11th International Winter Conference on Brain-Computer Interface, BCI 2023 - Virtual, Online, Korea, Republic of Duration: 2023 Feb 20 → 2023 Feb 22

Publication series

Name	International Winter Conference on Brain-Computer Interface, BCI
Volume	2023-February
ISSN (Print)	2572-7672

Conference

Conference	11th International Winter Conference on Brain-Computer Interface, BCI 2023
Country/Territory	Korea, Republic of
City	Virtual, Online
Period	23/2/20 → 23/2/22

Bibliographical note

Funding Information:
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program(Korea University)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00432, Development Of Non-invasive Integrated BCI SW Platform To Control Home Appliance And External Devices By User’s Thought Via AR/VR Interface), Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-02068, Artificial Intelligence Innovation Hub), and the BK21 four program through the National Research Foundation (NRF) funded by the Ministry of Education of Korea.

Publisher Copyright:
© 2023 IEEE.

Keywords

Brain-Computer Interface (BCI)
Electroencephalogram (EEG)
Sleep Stage Classification
Transformer.

ASJC Scopus subject areas

Artificial Intelligence
Human-Computer Interaction
Signal Processing

Access to Document

10.1109/BCI57258.2023.10078700

Cite this

Jung, K., Kim, M., & Chung, W. (2023). Epoch-level and Sequence-level Multi-Head Self-Attention-based Sleep Stage Classification. In 11th International Winter Conference on Brain-Computer Interface, BCI 2023 (International Winter Conference on Brain-Computer Interface, BCI; Vol. 2023-February). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BCI57258.2023.10078700

Epoch-level and Sequence-level Multi-Head Self-Attention-based Sleep Stage Classification. / Jung, Koohong; Kim, Moogyeong; Chung, Wonzoo.
11th International Winter Conference on Brain-Computer Interface, BCI 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (International Winter Conference on Brain-Computer Interface, BCI; Vol. 2023-February).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Jung, K, Kim, M & Chung, W 2023, Epoch-level and Sequence-level Multi-Head Self-Attention-based Sleep Stage Classification. in 11th International Winter Conference on Brain-Computer Interface, BCI 2023. International Winter Conference on Brain-Computer Interface, BCI, vol. 2023-February, Institute of Electrical and Electronics Engineers Inc., 11th International Winter Conference on Brain-Computer Interface, BCI 2023, Virtual, Online, Korea, Republic of, 23/2/20. https://doi.org/10.1109/BCI57258.2023.10078700

@inproceedings{6014c73776bc4aad924ede407a9ce233,

title = "Epoch-level and Sequence-level Multi-Head Self-Attention-based Sleep Stage Classification",

abstract = "Inthis paper, we propose a Transformer-based sleep stage classification method by effectively combining epoch-level and sequence-level multi-head self-attention with multiple outputs to aid the training. Most of the existing automated sleep staging methods utilized convolutional neural networks (CNNs) for feature extraction and recurrent neural networks (RNNs) or Transformer for capturing global dependencies. However, there are disadvantages when utilizing CNNs and RNNs (e.g., large parameter sizes, information loss). Therefore, Transformer-only architecture with epoch-level multi-head self-attention for capturing intra-epoch dependencies and sequence-level multihead self-attention for capturing inter-epoch dependencies in 30-second PSG signals is utilized. Furthermore, a sequence of learnable class tokens is deployed to store summary of each sleep epoch. As a result, three different outputs are produced, in which two are used as references during the backpropagation in training. Numerical simulation results on the Sleep-EDFX78 dataset confirm that the proposed method show improvements compared to the existing Transformer-based methods.",

keywords = "Brain-Computer Interface (BCI), Electroencephalogram (EEG), Sleep Stage Classification, Transformer.",

author = "Koohong Jung and Moogyeong Kim and Wonzoo Chung",

note = "Funding Information: This work was partly supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program(Korea University)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00432, Development Of Non-invasive Integrated BCI SW Platform To Control Home Appliance And External Devices By User{\textquoteright}s Thought Via AR/VR Interface), Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User{\textquoteright}s Intentions using Deep Learning), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-02068, Artificial Intelligence Innovation Hub), and the BK21 four program through the National Research Foundation (NRF) funded by the Ministry of Education of Korea. Publisher Copyright: {\textcopyright} 2023 IEEE.; 11th International Winter Conference on Brain-Computer Interface, BCI 2023 ; Conference date: 20-02-2023 Through 22-02-2023",

year = "2023",

doi = "10.1109/BCI57258.2023.10078700",

language = "English",

series = "International Winter Conference on Brain-Computer Interface, BCI",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "11th International Winter Conference on Brain-Computer Interface, BCI 2023",

}

TY - GEN

T1 - Epoch-level and Sequence-level Multi-Head Self-Attention-based Sleep Stage Classification

AU - Jung, Koohong

AU - Kim, Moogyeong

AU - Chung, Wonzoo

N1 - Funding Information: This work was partly supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program(Korea University)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00432, Development Of Non-invasive Integrated BCI SW Platform To Control Home Appliance And External Devices By User’s Thought Via AR/VR Interface), Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-02068, Artificial Intelligence Innovation Hub), and the BK21 four program through the National Research Foundation (NRF) funded by the Ministry of Education of Korea. Publisher Copyright: © 2023 IEEE.

PY - 2023

Y1 - 2023

N2 - Inthis paper, we propose a Transformer-based sleep stage classification method by effectively combining epoch-level and sequence-level multi-head self-attention with multiple outputs to aid the training. Most of the existing automated sleep staging methods utilized convolutional neural networks (CNNs) for feature extraction and recurrent neural networks (RNNs) or Transformer for capturing global dependencies. However, there are disadvantages when utilizing CNNs and RNNs (e.g., large parameter sizes, information loss). Therefore, Transformer-only architecture with epoch-level multi-head self-attention for capturing intra-epoch dependencies and sequence-level multihead self-attention for capturing inter-epoch dependencies in 30-second PSG signals is utilized. Furthermore, a sequence of learnable class tokens is deployed to store summary of each sleep epoch. As a result, three different outputs are produced, in which two are used as references during the backpropagation in training. Numerical simulation results on the Sleep-EDFX78 dataset confirm that the proposed method show improvements compared to the existing Transformer-based methods.

AB - Inthis paper, we propose a Transformer-based sleep stage classification method by effectively combining epoch-level and sequence-level multi-head self-attention with multiple outputs to aid the training. Most of the existing automated sleep staging methods utilized convolutional neural networks (CNNs) for feature extraction and recurrent neural networks (RNNs) or Transformer for capturing global dependencies. However, there are disadvantages when utilizing CNNs and RNNs (e.g., large parameter sizes, information loss). Therefore, Transformer-only architecture with epoch-level multi-head self-attention for capturing intra-epoch dependencies and sequence-level multihead self-attention for capturing inter-epoch dependencies in 30-second PSG signals is utilized. Furthermore, a sequence of learnable class tokens is deployed to store summary of each sleep epoch. As a result, three different outputs are produced, in which two are used as references during the backpropagation in training. Numerical simulation results on the Sleep-EDFX78 dataset confirm that the proposed method show improvements compared to the existing Transformer-based methods.

KW - Brain-Computer Interface (BCI)

KW - Electroencephalogram (EEG)

KW - Sleep Stage Classification

KW - Transformer.

UR - http://www.scopus.com/inward/record.url?scp=85152202223&partnerID=8YFLogxK

U2 - 10.1109/BCI57258.2023.10078700

DO - 10.1109/BCI57258.2023.10078700

M3 - Conference contribution

AN - SCOPUS:85152202223

T3 - International Winter Conference on Brain-Computer Interface, BCI

BT - 11th International Winter Conference on Brain-Computer Interface, BCI 2023

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 11th International Winter Conference on Brain-Computer Interface, BCI 2023

Y2 - 20 February 2023 through 22 February 2023

ER -

Epoch-level and Sequence-level Multi-Head Self-Attention-based Sleep Stage Classification

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this