Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training

Joongho Jo; Jongsun Park

doi:10.1109/ISOCC56007.2022.10031597

Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training

Joongho Jo, Jongsun Park

School of Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

Low-bit quantization of CNN training is highly needed for reducing the computational complexity of convolutional neural network (CNN) training. In CNN training, some of the classes can finish training early (reaches high accuracy in early training epochs) while other classes need more time (epochs) to finish training. This measure of training difficulty can be efficiently exploited for the mixed precision quantization to reduce the computational complexity of CNN training. In this paper, we present a training difficulty based mixed precision training approach, where easy-to-train classes are trained using low-bit quantization and the hard-to-train classes are trained using high bit quantization. The simulation results show that the proposed mixed precision training can achieve 1.33X improved compression ratio with the same accuracy compared to 8-bit (activations and weights) and 16-bit (gradients of activation and weight) uniform quantization training for ResNet-20 using the CIFAR-10 dataset.

Original language	English
Title of host publication	Proceedings - International SoC Design Conference 2022, ISOCC 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	372-373
Number of pages	2
ISBN (Electronic)	9781665459716
DOIs	https://doi.org/10.1109/ISOCC56007.2022.10031597
Publication status	Published - 2022
Event	19th International System-on-Chip Design Conference, ISOCC 2022 - Gangneung-si, Korea, Republic of Duration: 2022 Oct 19 → 2022 Oct 22

Publication series

Name	Proceedings - International SoC Design Conference 2022, ISOCC 2022

Conference

Conference	19th International System-on-Chip Design Conference, ISOCC 2022
Country/Territory	Korea, Republic of
City	Gangneung-si
Period	22/10/19 → 22/10/22

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea grant funded by the Korea government (NRF-2020R1A2C3014820).

Publisher Copyright:
© 2022 IEEE.

Keywords

Convolutional Neural Network (CNN)
Low-bit quantization training
Mixed precision training

ASJC Scopus subject areas

Artificial Intelligence
Computer Science Applications
Hardware and Architecture
Safety, Risk, Reliability and Quality

Access to Document

10.1109/ISOCC56007.2022.10031597

Cite this

Jo, J., & Park, J. (2022). Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training. In Proceedings - International SoC Design Conference 2022, ISOCC 2022 (pp. 372-373). (Proceedings - International SoC Design Conference 2022, ISOCC 2022). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ISOCC56007.2022.10031597

Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training. / Jo, Joongho; Park, Jongsun.
Proceedings - International SoC Design Conference 2022, ISOCC 2022. Institute of Electrical and Electronics Engineers Inc., 2022. p. 372-373 (Proceedings - International SoC Design Conference 2022, ISOCC 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Jo, J & Park, J 2022, Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training. in Proceedings - International SoC Design Conference 2022, ISOCC 2022. Proceedings - International SoC Design Conference 2022, ISOCC 2022, Institute of Electrical and Electronics Engineers Inc., pp. 372-373, 19th International System-on-Chip Design Conference, ISOCC 2022, Gangneung-si, Korea, Republic of, 22/10/19. https://doi.org/10.1109/ISOCC56007.2022.10031597

@inproceedings{51f70382196b454281f484993ae2bc41,

title = "Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training",

abstract = "Low-bit quantization of CNN training is highly needed for reducing the computational complexity of convolutional neural network (CNN) training. In CNN training, some of the classes can finish training early (reaches high accuracy in early training epochs) while other classes need more time (epochs) to finish training. This measure of training difficulty can be efficiently exploited for the mixed precision quantization to reduce the computational complexity of CNN training. In this paper, we present a training difficulty based mixed precision training approach, where easy-to-train classes are trained using low-bit quantization and the hard-to-train classes are trained using high bit quantization. The simulation results show that the proposed mixed precision training can achieve 1.33X improved compression ratio with the same accuracy compared to 8-bit (activations and weights) and 16-bit (gradients of activation and weight) uniform quantization training for ResNet-20 using the CIFAR-10 dataset.",

keywords = "Convolutional Neural Network (CNN), Low-bit quantization training, Mixed precision training",

author = "Joongho Jo and Jongsun Park",

note = "Funding Information: This work was supported by the National Research Foundation of Korea grant funded by the Korea government (NRF-2020R1A2C3014820). Publisher Copyright: {\textcopyright} 2022 IEEE.; 19th International System-on-Chip Design Conference, ISOCC 2022 ; Conference date: 19-10-2022 Through 22-10-2022",

year = "2022",

doi = "10.1109/ISOCC56007.2022.10031597",

language = "English",

series = "Proceedings - International SoC Design Conference 2022, ISOCC 2022",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "372--373",

booktitle = "Proceedings - International SoC Design Conference 2022, ISOCC 2022",

}

TY - GEN

T1 - Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training

AU - Jo, Joongho

AU - Park, Jongsun

PY - 2022

Y1 - 2022

N2 - Low-bit quantization of CNN training is highly needed for reducing the computational complexity of convolutional neural network (CNN) training. In CNN training, some of the classes can finish training early (reaches high accuracy in early training epochs) while other classes need more time (epochs) to finish training. This measure of training difficulty can be efficiently exploited for the mixed precision quantization to reduce the computational complexity of CNN training. In this paper, we present a training difficulty based mixed precision training approach, where easy-to-train classes are trained using low-bit quantization and the hard-to-train classes are trained using high bit quantization. The simulation results show that the proposed mixed precision training can achieve 1.33X improved compression ratio with the same accuracy compared to 8-bit (activations and weights) and 16-bit (gradients of activation and weight) uniform quantization training for ResNet-20 using the CIFAR-10 dataset.

AB - Low-bit quantization of CNN training is highly needed for reducing the computational complexity of convolutional neural network (CNN) training. In CNN training, some of the classes can finish training early (reaches high accuracy in early training epochs) while other classes need more time (epochs) to finish training. This measure of training difficulty can be efficiently exploited for the mixed precision quantization to reduce the computational complexity of CNN training. In this paper, we present a training difficulty based mixed precision training approach, where easy-to-train classes are trained using low-bit quantization and the hard-to-train classes are trained using high bit quantization. The simulation results show that the proposed mixed precision training can achieve 1.33X improved compression ratio with the same accuracy compared to 8-bit (activations and weights) and 16-bit (gradients of activation and weight) uniform quantization training for ResNet-20 using the CIFAR-10 dataset.

KW - Convolutional Neural Network (CNN)

KW - Low-bit quantization training

KW - Mixed precision training

UR - http://www.scopus.com/inward/record.url?scp=85148423183&partnerID=8YFLogxK

U2 - 10.1109/ISOCC56007.2022.10031597

DO - 10.1109/ISOCC56007.2022.10031597

M3 - Conference contribution

AN - SCOPUS:85148423183

T3 - Proceedings - International SoC Design Conference 2022, ISOCC 2022

SP - 372

EP - 373

BT - Proceedings - International SoC Design Conference 2022, ISOCC 2022

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th International System-on-Chip Design Conference, ISOCC 2022

Y2 - 19 October 2022 through 22 October 2022

ER -

Class Difficulty based Mixed Precision Quantization for Low Complexity CNN Training

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this