Quantized Sparse Training: A Unified Trainable Framework for Joint Pruning and Quantization in DNNs

Jun Hyung Park, Kang Min Kim, Sangkeun Lee

Research output: Contribution to journalArticlepeer-review

6 Citations (Scopus)

Abstract

Deep neural networks typically have extensive parameters and computational operations. Pruning and quantization techniques have been widely used to reduce the complexity of deep models. Both techniques can be jointly used for realizing significantly higher compression ratios. However, separate optimization processes and difficulties in choosing the hyperparameters limit the application of both the techniques simultaneously. In this study, we propose a novel compression framework, termed as quantized sparse training, that prunes and quantizes networks jointly in a unified training process. We integrate pruning and quantization into a gradient-based optimization process based on the straight-through estimator. Quantized sparse training enables us to simultaneously train, prune, and quantize a network from scratch. The empirical results validate the superiority of the proposed methodology over the recent state-of-the-art baselines with respect to both the model size and accuracy. Specifically, quantized sparse training achieves a 135 KB model size in the case of VGG16, without any accuracy degradation, which is 40% of the model size feasible based on the state-of-the-art pruning and quantization approach.

Original languageEnglish
Article number3524066
JournalACM Transactions on Embedded Computing Systems
Volume21
Issue number5
DOIs
Publication statusPublished - 2022 Oct 8

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (Grant No. 2021R1A2C3010430), the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (Grant No. 2020R1A4A1018309), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (Grant No. 2019-0-00079, Artificial Intelligence Graduate School Program (Korea University)).

Publisher Copyright:
© 2022 Copyright held by the owner/author(s). Publication rights licensed to ACM.

Keywords

  • Deep learning
  • joint pruning and quantization
  • model compression
  • neural network

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture

Fingerprint

Dive into the research topics of 'Quantized Sparse Training: A Unified Trainable Framework for Joint Pruning and Quantization in DNNs'. Together they form a unique fingerprint.

Cite this