Sleep-CMKD: Self-Attention CNN/Transformer Cross-Model Knowledge Distillation for Automatic Sleep Staging

Hyounggyu Kim; Moogyeong Kim; Wonzoo Chung

doi:10.1109/BCI57258.2023.10078646

Sleep-CMKD: Self-Attention CNN/Transformer Cross-Model Knowledge Distillation for Automatic Sleep Staging

Hyounggyu Kim, Moogyeong Kim, Wonzoo Chung

Department of Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we propose single-channel cross-model knowledge distillation(CMKD) method between convolutional neural networks-based and transformer-based models for automatic sleep staging. In sleep staging, few works proposed to distill knowledge from additional sleep dataset or multi-channel polysomnogram requiring manual scoring effort of human experts and limiting the home application senarios. Cross-model knowledge distillation avoids these additional efforts and limitations by distilling inductive biases between models with different structures. Experiments on Sleep-EDFX-78 dataset confirm that the proposed method improves sleep stage classification accuracy of transformer-based model by 1.7%.

Original language	English
Title of host publication	11th International Winter Conference on Brain-Computer Interface, BCI 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665464444
DOIs	https://doi.org/10.1109/BCI57258.2023.10078646
Publication status	Published - 2023
Event	11th International Winter Conference on Brain-Computer Interface, BCI 2023 - Virtual, Online, Korea, Republic of Duration: 2023 Feb 20 → 2023 Feb 22

Publication series

Name	International Winter Conference on Brain-Computer Interface, BCI
Volume	2023-February
ISSN (Print)	2572-7672

Conference

Conference	11th International Winter Conference on Brain-Computer Interface, BCI 2023
Country/Territory	Korea, Republic of
City	Virtual, Online
Period	23/2/20 → 23/2/22

Bibliographical note

Funding Information:
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program(Korea University)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00432, Development Of Non-invasive Integrated BCI SW Platform To Control Home Appliance And External Devices By User’s Thought Via AR/VR Interface), Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-02068, Artificial Intelligence Innovation Hub), and the BK21 four program through the National Research Foundation (NRF) funded by the Ministry of Education of Korea.

Publisher Copyright:
© 2023 IEEE.

Keywords

Automatic Sleep Staging
Electroencephalogram (EEG)
Knowledge Distillation
Transformer

ASJC Scopus subject areas

Artificial Intelligence
Human-Computer Interaction
Signal Processing

Access to Document

10.1109/BCI57258.2023.10078646

Cite this

Kim, H., Kim, M., & Chung, W. (2023). Sleep-CMKD: Self-Attention CNN/Transformer Cross-Model Knowledge Distillation for Automatic Sleep Staging. In 11th International Winter Conference on Brain-Computer Interface, BCI 2023 (International Winter Conference on Brain-Computer Interface, BCI; Vol. 2023-February). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BCI57258.2023.10078646

Sleep-CMKD: Self-Attention CNN/Transformer Cross-Model Knowledge Distillation for Automatic Sleep Staging. / Kim, Hyounggyu; Kim, Moogyeong; Chung, Wonzoo.
11th International Winter Conference on Brain-Computer Interface, BCI 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (International Winter Conference on Brain-Computer Interface, BCI; Vol. 2023-February).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kim, H, Kim, M & Chung, W 2023, Sleep-CMKD: Self-Attention CNN/Transformer Cross-Model Knowledge Distillation for Automatic Sleep Staging. in 11th International Winter Conference on Brain-Computer Interface, BCI 2023. International Winter Conference on Brain-Computer Interface, BCI, vol. 2023-February, Institute of Electrical and Electronics Engineers Inc., 11th International Winter Conference on Brain-Computer Interface, BCI 2023, Virtual, Online, Korea, Republic of, 23/2/20. https://doi.org/10.1109/BCI57258.2023.10078646

Kim H, Kim M, Chung W. Sleep-CMKD: Self-Attention CNN/Transformer Cross-Model Knowledge Distillation for Automatic Sleep Staging. In 11th International Winter Conference on Brain-Computer Interface, BCI 2023. Institute of Electrical and Electronics Engineers Inc. 2023. (International Winter Conference on Brain-Computer Interface, BCI). doi: 10.1109/BCI57258.2023.10078646

@inproceedings{54985caa6401477fbf28ad6d21f2e402,

title = "Sleep-CMKD: Self-Attention CNN/Transformer Cross-Model Knowledge Distillation for Automatic Sleep Staging",

abstract = "In this paper, we propose single-channel cross-model knowledge distillation(CMKD) method between convolutional neural networks-based and transformer-based models for automatic sleep staging. In sleep staging, few works proposed to distill knowledge from additional sleep dataset or multi-channel polysomnogram requiring manual scoring effort of human experts and limiting the home application senarios. Cross-model knowledge distillation avoids these additional efforts and limitations by distilling inductive biases between models with different structures. Experiments on Sleep-EDFX-78 dataset confirm that the proposed method improves sleep stage classification accuracy of transformer-based model by 1.7%.",

keywords = "Automatic Sleep Staging, Electroencephalogram (EEG), Knowledge Distillation, Transformer",

author = "Hyounggyu Kim and Moogyeong Kim and Wonzoo Chung",

note = "Funding Information: This work was partly supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program(Korea University)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00432, Development Of Non-invasive Integrated BCI SW Platform To Control Home Appliance And External Devices By User{\textquoteright}s Thought Via AR/VR Interface), Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User{\textquoteright}s Intentions using Deep Learning), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-02068, Artificial Intelligence Innovation Hub), and the BK21 four program through the National Research Foundation (NRF) funded by the Ministry of Education of Korea. Publisher Copyright: {\textcopyright} 2023 IEEE.; 11th International Winter Conference on Brain-Computer Interface, BCI 2023 ; Conference date: 20-02-2023 Through 22-02-2023",

year = "2023",

doi = "10.1109/BCI57258.2023.10078646",

language = "English",

series = "International Winter Conference on Brain-Computer Interface, BCI",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "11th International Winter Conference on Brain-Computer Interface, BCI 2023",

}

TY - GEN

T1 - Sleep-CMKD

T2 - 11th International Winter Conference on Brain-Computer Interface, BCI 2023

AU - Kim, Hyounggyu

AU - Kim, Moogyeong

AU - Chung, Wonzoo

N1 - Funding Information: This work was partly supported by Institute of Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2019-0-00079, Artificial Intelligence Graduate School Program(Korea University)), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00432, Development Of Non-invasive Integrated BCI SW Platform To Control Home Appliance And External Devices By User’s Thought Via AR/VR Interface), Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2017-0-00451, Development of BCI based Brain and Cognitive Computing Technology for Recognizing User’s Intentions using Deep Learning), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-02068, Artificial Intelligence Innovation Hub), and the BK21 four program through the National Research Foundation (NRF) funded by the Ministry of Education of Korea. Publisher Copyright: © 2023 IEEE.

PY - 2023

Y1 - 2023

N2 - In this paper, we propose single-channel cross-model knowledge distillation(CMKD) method between convolutional neural networks-based and transformer-based models for automatic sleep staging. In sleep staging, few works proposed to distill knowledge from additional sleep dataset or multi-channel polysomnogram requiring manual scoring effort of human experts and limiting the home application senarios. Cross-model knowledge distillation avoids these additional efforts and limitations by distilling inductive biases between models with different structures. Experiments on Sleep-EDFX-78 dataset confirm that the proposed method improves sleep stage classification accuracy of transformer-based model by 1.7%.

AB - In this paper, we propose single-channel cross-model knowledge distillation(CMKD) method between convolutional neural networks-based and transformer-based models for automatic sleep staging. In sleep staging, few works proposed to distill knowledge from additional sleep dataset or multi-channel polysomnogram requiring manual scoring effort of human experts and limiting the home application senarios. Cross-model knowledge distillation avoids these additional efforts and limitations by distilling inductive biases between models with different structures. Experiments on Sleep-EDFX-78 dataset confirm that the proposed method improves sleep stage classification accuracy of transformer-based model by 1.7%.

KW - Automatic Sleep Staging

KW - Electroencephalogram (EEG)

KW - Knowledge Distillation

KW - Transformer

UR - http://www.scopus.com/inward/record.url?scp=85152203061&partnerID=8YFLogxK

U2 - 10.1109/BCI57258.2023.10078646

DO - 10.1109/BCI57258.2023.10078646

M3 - Conference contribution

AN - SCOPUS:85152203061

T3 - International Winter Conference on Brain-Computer Interface, BCI

BT - 11th International Winter Conference on Brain-Computer Interface, BCI 2023

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 20 February 2023 through 22 February 2023

ER -

Sleep-CMKD: Self-Attention CNN/Transformer Cross-Model Knowledge Distillation for Automatic Sleep Staging

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this