Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks

Kyungchul Lee; Jongsun Park

doi:10.1109/ISOCC56007.2022.10031502

Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks

Kyungchul Lee, Jongsun Park

School of Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Although deep neural networks (DNNs) show excellent performance in the image processing field, a massive amount of computation and memory access makes it difficult to deploy DNNs on mobile devices. To reduce the computation and the memory access, quantization shows great results. However, it is challenging to quantize the network into low bit-width without significant accuracy degradation. In this paper, we propose Clipped Quantization Aware Training (CQAT) to decrease the accuracy drop during the low bit-width quantization aware training. In the proposed CQAT, the original values are first quantized into 8-bit and then the quantized values are clipped so that only 4-bit for activations and 5-bit for weights are used. With the proposed technique, ResNet-18 for the CIFAR-100 dataset using 5-bit weight and 4-bit activation shows an accuracy drop of only 0.96% compared to the network using full precision weight and activation.

Original language	English
Title of host publication	Proceedings - International SoC Design Conference 2022, ISOCC 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	370-371
Number of pages	2
ISBN (Electronic)	9781665459716
DOIs	https://doi.org/10.1109/ISOCC56007.2022.10031502
Publication status	Published - 2022
Event	19th International System-on-Chip Design Conference, ISOCC 2022 - Gangneung-si, Korea, Republic of Duration: 2022 Oct 19 → 2022 Oct 22

Publication series

Name	Proceedings - International SoC Design Conference 2022, ISOCC 2022

Conference

Conference	19th International System-on-Chip Design Conference, ISOCC 2022
Country/Territory	Korea, Republic of
City	Gangneung-si
Period	22/10/19 → 22/10/22

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea grant funded by the Korea government (NRF-2020R1A2C3014820)

Publisher Copyright:
© 2022 IEEE.

Keywords

deep neural network
quantization
quantization aware training

ASJC Scopus subject areas

Artificial Intelligence
Computer Science Applications
Hardware and Architecture
Safety, Risk, Reliability and Quality

Access to Document

10.1109/ISOCC56007.2022.10031502

Cite this

Lee, K., & Park, J. (2022). Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks. In Proceedings - International SoC Design Conference 2022, ISOCC 2022 (pp. 370-371). (Proceedings - International SoC Design Conference 2022, ISOCC 2022). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISOCC56007.2022.10031502

Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks. / Lee, Kyungchul; Park, Jongsun.
Proceedings - International SoC Design Conference 2022, ISOCC 2022. Institute of Electrical and Electronics Engineers Inc., 2022. p. 370-371 (Proceedings - International SoC Design Conference 2022, ISOCC 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lee, K & Park, J 2022, Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks. in Proceedings - International SoC Design Conference 2022, ISOCC 2022. Proceedings - International SoC Design Conference 2022, ISOCC 2022, Institute of Electrical and Electronics Engineers Inc., pp. 370-371, 19th International System-on-Chip Design Conference, ISOCC 2022, Gangneung-si, Korea, Republic of, 22/10/19. https://doi.org/10.1109/ISOCC56007.2022.10031502

Lee K, Park J. Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks. In Proceedings - International SoC Design Conference 2022, ISOCC 2022. Institute of Electrical and Electronics Engineers Inc. 2022. p. 370-371. (Proceedings - International SoC Design Conference 2022, ISOCC 2022). doi: 10.1109/ISOCC56007.2022.10031502

@inproceedings{b525aac05ad54f8c86f01516d6e080b6,

title = "Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks",

abstract = "Although deep neural networks (DNNs) show excellent performance in the image processing field, a massive amount of computation and memory access makes it difficult to deploy DNNs on mobile devices. To reduce the computation and the memory access, quantization shows great results. However, it is challenging to quantize the network into low bit-width without significant accuracy degradation. In this paper, we propose Clipped Quantization Aware Training (CQAT) to decrease the accuracy drop during the low bit-width quantization aware training. In the proposed CQAT, the original values are first quantized into 8-bit and then the quantized values are clipped so that only 4-bit for activations and 5-bit for weights are used. With the proposed technique, ResNet-18 for the CIFAR-100 dataset using 5-bit weight and 4-bit activation shows an accuracy drop of only 0.96% compared to the network using full precision weight and activation.",

keywords = "deep neural network, quantization, quantization aware training",

author = "Kyungchul Lee and Jongsun Park",

note = "Funding Information: This work was supported by the National Research Foundation of Korea grant funded by the Korea government (NRF-2020R1A2C3014820) Publisher Copyright: {\textcopyright} 2022 IEEE.; 19th International System-on-Chip Design Conference, ISOCC 2022 ; Conference date: 19-10-2022 Through 22-10-2022",

year = "2022",

doi = "10.1109/ISOCC56007.2022.10031502",

language = "English",

series = "Proceedings - International SoC Design Conference 2022, ISOCC 2022",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "370--371",

booktitle = "Proceedings - International SoC Design Conference 2022, ISOCC 2022",

}

TY - GEN

T1 - Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks

AU - Lee, Kyungchul

AU - Park, Jongsun

PY - 2022

Y1 - 2022

N2 - Although deep neural networks (DNNs) show excellent performance in the image processing field, a massive amount of computation and memory access makes it difficult to deploy DNNs on mobile devices. To reduce the computation and the memory access, quantization shows great results. However, it is challenging to quantize the network into low bit-width without significant accuracy degradation. In this paper, we propose Clipped Quantization Aware Training (CQAT) to decrease the accuracy drop during the low bit-width quantization aware training. In the proposed CQAT, the original values are first quantized into 8-bit and then the quantized values are clipped so that only 4-bit for activations and 5-bit for weights are used. With the proposed technique, ResNet-18 for the CIFAR-100 dataset using 5-bit weight and 4-bit activation shows an accuracy drop of only 0.96% compared to the network using full precision weight and activation.

AB - Although deep neural networks (DNNs) show excellent performance in the image processing field, a massive amount of computation and memory access makes it difficult to deploy DNNs on mobile devices. To reduce the computation and the memory access, quantization shows great results. However, it is challenging to quantize the network into low bit-width without significant accuracy degradation. In this paper, we propose Clipped Quantization Aware Training (CQAT) to decrease the accuracy drop during the low bit-width quantization aware training. In the proposed CQAT, the original values are first quantized into 8-bit and then the quantized values are clipped so that only 4-bit for activations and 5-bit for weights are used. With the proposed technique, ResNet-18 for the CIFAR-100 dataset using 5-bit weight and 4-bit activation shows an accuracy drop of only 0.96% compared to the network using full precision weight and activation.

KW - deep neural network

KW - quantization

KW - quantization aware training

UR - http://www.scopus.com/inward/record.url?scp=85148466911&partnerID=8YFLogxK

U2 - 10.1109/ISOCC56007.2022.10031502

DO - 10.1109/ISOCC56007.2022.10031502

M3 - Conference contribution

AN - SCOPUS:85148466911

T3 - Proceedings - International SoC Design Conference 2022, ISOCC 2022

SP - 370

EP - 371

BT - Proceedings - International SoC Design Conference 2022, ISOCC 2022

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th International System-on-Chip Design Conference, ISOCC 2022

Y2 - 19 October 2022 through 22 October 2022

ER -

Clipped Quantization Aware Training for Hardware Friendly Implementation of Image Classification Networks

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this