Channel-Wise Activation Map Pruning using Max-Pool for Reducing Memory Accesses

Han Cho; Jongsun Park

doi:10.1109/ISOCC56007.2022.10031452

Channel-Wise Activation Map Pruning using Max-Pool for Reducing Memory Accesses

Han Cho, Jongsun Park

School of Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

While neural network pruning can reduce the amount of data transfers, pruning techniques such as fine-grained pruning cannot be efficiently implemented as indexing overhead is high. In order to design a hardware friendly pruning technique, structured pruning, which removes groups of data together to minimize the indexing overhead, is highly required. In this paper, to reduce the overall number of main memory accesses when implementing convolutional neural network (CNN) accelerators, we propose a hardware friendly L2-norm based structured channel-wise activation pruning using max-pooling. Max-pooling is typically deployed in CNN to decrease the height and width of CNN activation maps. The simulation results of the proposed technique shows that the activation maps of ResNet20 and ResNet56 can be reduced to 48% and 51%, respectively, with less than 1% accuracy degradation on CIFAR-10.

Original language	English
Title of host publication	Proceedings - International SoC Design Conference 2022, ISOCC 2022
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	71-72
Number of pages	2
ISBN (Electronic)	9781665459716
DOIs	https://doi.org/10.1109/ISOCC56007.2022.10031452
Publication status	Published - 2022
Event	19th International System-on-Chip Design Conference, ISOCC 2022 - Gangneung-si, Korea, Republic of Duration: 2022 Oct 19 → 2022 Oct 22

Publication series

Name	Proceedings - International SoC Design Conference 2022, ISOCC 2022

Conference

Conference	19th International System-on-Chip Design Conference, ISOCC 2022
Country/Territory	Korea, Republic of
City	Gangneung-si
Period	22/10/19 → 22/10/22

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea grant funded by the Korea government (NRF-2020R1A2C3014820)

Publisher Copyright:
© 2022 IEEE.

Keywords

activation compression
activation pruning
Convolutional neural network
L2-norm
max-pool

ASJC Scopus subject areas

Artificial Intelligence
Computer Science Applications
Hardware and Architecture
Safety, Risk, Reliability and Quality

Access to Document

10.1109/ISOCC56007.2022.10031452

Cite this

Channel-Wise Activation Map Pruning using Max-Pool for Reducing Memory Accesses. / Cho, Han; Park, Jongsun.
Proceedings - International SoC Design Conference 2022, ISOCC 2022. Institute of Electrical and Electronics Engineers Inc., 2022. p. 71-72 (Proceedings - International SoC Design Conference 2022, ISOCC 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Cho, H & Park, J 2022, Channel-Wise Activation Map Pruning using Max-Pool for Reducing Memory Accesses. in Proceedings - International SoC Design Conference 2022, ISOCC 2022. Proceedings - International SoC Design Conference 2022, ISOCC 2022, Institute of Electrical and Electronics Engineers Inc., pp. 71-72, 19th International System-on-Chip Design Conference, ISOCC 2022, Gangneung-si, Korea, Republic of, 22/10/19. https://doi.org/10.1109/ISOCC56007.2022.10031452

@inproceedings{6a63586324264306a48bcf94e9048806,

title = "Channel-Wise Activation Map Pruning using Max-Pool for Reducing Memory Accesses",

abstract = "While neural network pruning can reduce the amount of data transfers, pruning techniques such as fine-grained pruning cannot be efficiently implemented as indexing overhead is high. In order to design a hardware friendly pruning technique, structured pruning, which removes groups of data together to minimize the indexing overhead, is highly required. In this paper, to reduce the overall number of main memory accesses when implementing convolutional neural network (CNN) accelerators, we propose a hardware friendly L2-norm based structured channel-wise activation pruning using max-pooling. Max-pooling is typically deployed in CNN to decrease the height and width of CNN activation maps. The simulation results of the proposed technique shows that the activation maps of ResNet20 and ResNet56 can be reduced to 48% and 51%, respectively, with less than 1% accuracy degradation on CIFAR-10.",

keywords = "activation compression, activation pruning, Convolutional neural network, L2-norm, max-pool",

author = "Han Cho and Jongsun Park",

note = "Funding Information: This work was supported by the National Research Foundation of Korea grant funded by the Korea government (NRF-2020R1A2C3014820) Publisher Copyright: {\textcopyright} 2022 IEEE.; 19th International System-on-Chip Design Conference, ISOCC 2022 ; Conference date: 19-10-2022 Through 22-10-2022",

year = "2022",

doi = "10.1109/ISOCC56007.2022.10031452",

language = "English",

series = "Proceedings - International SoC Design Conference 2022, ISOCC 2022",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "71--72",

booktitle = "Proceedings - International SoC Design Conference 2022, ISOCC 2022",

}

TY - GEN

T1 - Channel-Wise Activation Map Pruning using Max-Pool for Reducing Memory Accesses

AU - Cho, Han

AU - Park, Jongsun

PY - 2022

Y1 - 2022

N2 - While neural network pruning can reduce the amount of data transfers, pruning techniques such as fine-grained pruning cannot be efficiently implemented as indexing overhead is high. In order to design a hardware friendly pruning technique, structured pruning, which removes groups of data together to minimize the indexing overhead, is highly required. In this paper, to reduce the overall number of main memory accesses when implementing convolutional neural network (CNN) accelerators, we propose a hardware friendly L2-norm based structured channel-wise activation pruning using max-pooling. Max-pooling is typically deployed in CNN to decrease the height and width of CNN activation maps. The simulation results of the proposed technique shows that the activation maps of ResNet20 and ResNet56 can be reduced to 48% and 51%, respectively, with less than 1% accuracy degradation on CIFAR-10.

AB - While neural network pruning can reduce the amount of data transfers, pruning techniques such as fine-grained pruning cannot be efficiently implemented as indexing overhead is high. In order to design a hardware friendly pruning technique, structured pruning, which removes groups of data together to minimize the indexing overhead, is highly required. In this paper, to reduce the overall number of main memory accesses when implementing convolutional neural network (CNN) accelerators, we propose a hardware friendly L2-norm based structured channel-wise activation pruning using max-pooling. Max-pooling is typically deployed in CNN to decrease the height and width of CNN activation maps. The simulation results of the proposed technique shows that the activation maps of ResNet20 and ResNet56 can be reduced to 48% and 51%, respectively, with less than 1% accuracy degradation on CIFAR-10.

KW - activation compression

KW - activation pruning

KW - Convolutional neural network

KW - L2-norm

KW - max-pool

UR - http://www.scopus.com/inward/record.url?scp=85148425651&partnerID=8YFLogxK

U2 - 10.1109/ISOCC56007.2022.10031452

DO - 10.1109/ISOCC56007.2022.10031452

M3 - Conference contribution

AN - SCOPUS:85148425651

T3 - Proceedings - International SoC Design Conference 2022, ISOCC 2022

SP - 71

EP - 72

BT - Proceedings - International SoC Design Conference 2022, ISOCC 2022

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th International System-on-Chip Design Conference, ISOCC 2022

Y2 - 19 October 2022 through 22 October 2022

ER -

Channel-Wise Activation Map Pruning using Max-Pool for Reducing Memory Accesses

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this