Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks

Kyungchul Lee; Sunghyun Choi; Dongwoo Lew; Jongsun Park

doi:10.1109/ITC-CSCC52171.2021.9501427

Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks

Kyungchul Lee, Sunghyun Choi, Dongwoo Lew, Jongsun Park

School of Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we present spiking neural network (SNN) conversion technique optimized for converting low bit-width artificial neural networks (ANN) trained with quantization aware training (QAT). Conventional conversion technique suffers significant accuracy drop on QAT ANNs due to different activation function used for QAT ANNs. To minimize such accuracy drop of the conventional conversion, the proposed technique uses Spike-Norm Skip, which selectively applies threshold balancing. In addition, subtraction based reset is used to further reduce accuracy degradation. The proposed conversion technique achieves an accuracy of 89.92% (0.68% drop) with a 5-bit weight on CIFAR-10 using VGG-16.

Original language	English
Title of host publication	2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665435536
DOIs	https://doi.org/10.1109/ITC-CSCC52171.2021.9501427
Publication status	Published - 2021 Jun 27
Event	36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 - Jeju, Korea, Republic of Duration: 2021 Jun 27 → 2021 Jun 30

Publication series

Name	2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

Conference

Conference	36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021
Country/Territory	Korea, Republic of
City	Jeju
Period	21/6/27 → 21/6/30

Bibliographical note

Funding Information:
This work was supported in part by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2021-2018-0-01433) supervised by the IITP(Institute for Information & communications Technology Promotion), and in part by the Industrial Strategic Technology Development Program(10077445, Development of SoC technology based on Spiking Neural Cell for smart mobile and IoT Devices) funded By the Ministry of Trade, Industry & Energy(MOTIE, Korea)

Publisher Copyright:
© 2021 IEEE.

Keywords

ANN-SNN conversion
quantization aware training
spiking neural networks

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications
Hardware and Architecture
Information Systems
Electrical and Electronic Engineering

Access to Document

10.1109/ITC-CSCC52171.2021.9501427

Cite this

Lee, K., Choi, S., Lew, D., & Park, J. (2021). Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks. In 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 Article 9501427 (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ITC-CSCC52171.2021.9501427

Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks. / Lee, Kyungchul; Choi, Sunghyun; Lew, Dongwoo et al.
2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc., 2021. 9501427 (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lee, K, Choi, S, Lew, D & Park, J 2021, Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks. in 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021., 9501427, 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021, Institute of Electrical and Electronics Engineers Inc., 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021, Jeju, Korea, Republic of, 21/6/27. https://doi.org/10.1109/ITC-CSCC52171.2021.9501427

Lee K, Choi S, Lew D, Park J. Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks. In 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc. 2021. 9501427. (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021). doi: 10.1109/ITC-CSCC52171.2021.9501427

Lee, Kyungchul ; Choi, Sunghyun ; Lew, Dongwoo et al. / Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks. 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc., 2021. (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021).

@inproceedings{6f6da8b446354172912d270d56828195,

title = "Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks",

abstract = "In this paper, we present spiking neural network (SNN) conversion technique optimized for converting low bit-width artificial neural networks (ANN) trained with quantization aware training (QAT). Conventional conversion technique suffers significant accuracy drop on QAT ANNs due to different activation function used for QAT ANNs. To minimize such accuracy drop of the conventional conversion, the proposed technique uses Spike-Norm Skip, which selectively applies threshold balancing. In addition, subtraction based reset is used to further reduce accuracy degradation. The proposed conversion technique achieves an accuracy of 89.92% (0.68% drop) with a 5-bit weight on CIFAR-10 using VGG-16. ",

keywords = "ANN-SNN conversion, quantization aware training, spiking neural networks",

author = "Kyungchul Lee and Sunghyun Choi and Dongwoo Lew and Jongsun Park",

note = "Funding Information: This work was supported in part by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2021-2018-0-01433) supervised by the IITP(Institute for Information & communications Technology Promotion), and in part by the Industrial Strategic Technology Development Program(10077445, Development of SoC technology based on Spiking Neural Cell for smart mobile and IoT Devices) funded By the Ministry of Trade, Industry & Energy(MOTIE, Korea) Publisher Copyright: {\textcopyright} 2021 IEEE.; 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 ; Conference date: 27-06-2021 Through 30-06-2021",

year = "2021",

month = jun,

day = "27",

doi = "10.1109/ITC-CSCC52171.2021.9501427",

language = "English",

series = "2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021",

}

TY - GEN

T1 - Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks

AU - Lee, Kyungchul

AU - Choi, Sunghyun

AU - Lew, Dongwoo

AU - Park, Jongsun

N1 - Funding Information: This work was supported in part by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program(IITP-2021-2018-0-01433) supervised by the IITP(Institute for Information & communications Technology Promotion), and in part by the Industrial Strategic Technology Development Program(10077445, Development of SoC technology based on Spiking Neural Cell for smart mobile and IoT Devices) funded By the Ministry of Trade, Industry & Energy(MOTIE, Korea) Publisher Copyright: © 2021 IEEE.

PY - 2021/6/27

Y1 - 2021/6/27

N2 - In this paper, we present spiking neural network (SNN) conversion technique optimized for converting low bit-width artificial neural networks (ANN) trained with quantization aware training (QAT). Conventional conversion technique suffers significant accuracy drop on QAT ANNs due to different activation function used for QAT ANNs. To minimize such accuracy drop of the conventional conversion, the proposed technique uses Spike-Norm Skip, which selectively applies threshold balancing. In addition, subtraction based reset is used to further reduce accuracy degradation. The proposed conversion technique achieves an accuracy of 89.92% (0.68% drop) with a 5-bit weight on CIFAR-10 using VGG-16.

AB - In this paper, we present spiking neural network (SNN) conversion technique optimized for converting low bit-width artificial neural networks (ANN) trained with quantization aware training (QAT). Conventional conversion technique suffers significant accuracy drop on QAT ANNs due to different activation function used for QAT ANNs. To minimize such accuracy drop of the conventional conversion, the proposed technique uses Spike-Norm Skip, which selectively applies threshold balancing. In addition, subtraction based reset is used to further reduce accuracy degradation. The proposed conversion technique achieves an accuracy of 89.92% (0.68% drop) with a 5-bit weight on CIFAR-10 using VGG-16.

KW - ANN-SNN conversion

KW - quantization aware training

KW - spiking neural networks

UR - http://www.scopus.com/inward/record.url?scp=85113973664&partnerID=8YFLogxK

U2 - 10.1109/ITC-CSCC52171.2021.9501427

DO - 10.1109/ITC-CSCC52171.2021.9501427

M3 - Conference contribution

AN - SCOPUS:85113973664

T3 - 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

BT - 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

Y2 - 27 June 2021 through 30 June 2021

ER -

Optimization Techniques for Conversion of Quantization Aware Trained Deep Neural Networks to Lightweight Spiking Neural Networks

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this