Low-bit quantization of CNN training is highly needed for reducing the computational complexity of convolutional neural network (CNN) training. In CNN training, some of the classes can finish training early (reaches high accuracy in early training epochs) while other classes need more time (epochs) to finish training. This measure of training difficulty can be efficiently exploited for the mixed precision quantization to reduce the computational complexity of CNN training. In this paper, we present a training difficulty based mixed precision training approach, where easy-to-train classes are trained using low-bit quantization and the hard-to-train classes are trained using high bit quantization. The simulation results show that the proposed mixed precision training can achieve 1.33X improved compression ratio with the same accuracy compared to 8-bit (activations and weights) and 16-bit (gradients of activation and weight) uniform quantization training for ResNet-20 using the CIFAR-10 dataset.
|Title of host publication||Proceedings - International SoC Design Conference 2022, ISOCC 2022|
|Publisher||Institute of Electrical and Electronics Engineers Inc.|
|Number of pages||2|
|Publication status||Published - 2022|
|Event||19th International System-on-Chip Design Conference, ISOCC 2022 - Gangneung-si, Korea, Republic of|
Duration: 2022 Oct 19 → 2022 Oct 22
|Name||Proceedings - International SoC Design Conference 2022, ISOCC 2022|
|Conference||19th International System-on-Chip Design Conference, ISOCC 2022|
|Country/Territory||Korea, Republic of|
|Period||22/10/19 → 22/10/22|
Bibliographical noteFunding Information:
This work was supported by the National Research Foundation of Korea grant funded by the Korea government (NRF-2020R1A2C3014820).
© 2022 IEEE.
- Convolutional Neural Network (CNN)
- Low-bit quantization training
- Mixed precision training
ASJC Scopus subject areas
- Artificial Intelligence
- Computer Science Applications
- Hardware and Architecture
- Safety, Risk, Reliability and Quality