Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells

Seok Young Kim; Chang Hyun Kim; Seon Wook Kim

doi:10.1109/ITC-CSCC52171.2021.9553004

Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells

Seok Young Kim, Chang Hyun Kim, Seon Wook Kim

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The characteristics of recurrent and functional reduction in RNNs (Recurrent Neural Networks) may cause an inefficient execution in conventional hardware such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit). Their large number of recurrent data prevents the CPU from exploiting their locality, while the reduction feature limits the GPU parallelism. As a result, several dedicated hardware for efficient RNN execution has been proposed recently. In this paper, we propose a processing element that performs primary operations (i.e., multiplication and accumulation operation and activation function operation) of LSTM (Long Short-Term Memory), the most dominant model of RNN We implemented the processing element on an FPGA (Field-Programmable Gate Array) and verified the accuracy by measuring the BLEU score (Bilingual Evaluation Understudy score) of the PyTorch-based LSTM application. It was decreased by 2.3% compared to the CPU result.

Original language	English
Title of host publication	2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781665435536
DOIs	https://doi.org/10.1109/ITC-CSCC52171.2021.9553004
Publication status	Published - 2021 Jun 27
Event	36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 - Jeju, Korea, Republic of Duration: 2021 Jun 27 → 2021 Jun 30

Publication series

Name	2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021
Volume	2021-January

Conference

Conference	36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021
Country/Territory	Korea, Republic of
City	Jeju
Period	21/6/27 → 21/6/30

Bibliographical note

Funding Information:
This work was supported in part by SK Hynix Inc.

Publisher Copyright:
© 2021 IEEE.

Keywords

Bfloat16
Floating-point MAC unit
Long Short-Term Memory
Recurrent Neural Networks

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications
Hardware and Architecture
Information Systems
Electrical and Electronic Engineering

Access to Document

10.1109/ITC-CSCC52171.2021.9553004

Cite this

Kim, S. Y., Kim, C. H., & Kim, S. W. (2021). Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells. In 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021; Vol. 2021-January). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ITC-CSCC52171.2021.9553004

Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells. / Kim, Seok Young; Kim, Chang Hyun; Kim, Seon Wook.
2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc., 2021. (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021; Vol. 2021-January).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kim, SY, Kim, CH & Kim, SW 2021, Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells. in 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021, vol. 2021-January, Institute of Electrical and Electronics Engineers Inc., 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021, Jeju, Korea, Republic of, 21/6/27. https://doi.org/10.1109/ITC-CSCC52171.2021.9553004

Kim SY, Kim CH, Kim SW. Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells. In 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc. 2021. (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021). doi: 10.1109/ITC-CSCC52171.2021.9553004

Kim, Seok Young ; Kim, Chang Hyun ; Kim, Seon Wook. / Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells. 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021. Institute of Electrical and Electronics Engineers Inc., 2021. (2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021).

@inproceedings{0f894f5ec5a646a4b8365db85fc98320,

title = "Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells",

abstract = "The characteristics of recurrent and functional reduction in RNNs (Recurrent Neural Networks) may cause an inefficient execution in conventional hardware such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit). Their large number of recurrent data prevents the CPU from exploiting their locality, while the reduction feature limits the GPU parallelism. As a result, several dedicated hardware for efficient RNN execution has been proposed recently. In this paper, we propose a processing element that performs primary operations (i.e., multiplication and accumulation operation and activation function operation) of LSTM (Long Short-Term Memory), the most dominant model of RNN We implemented the processing element on an FPGA (Field-Programmable Gate Array) and verified the accuracy by measuring the BLEU score (Bilingual Evaluation Understudy score) of the PyTorch-based LSTM application. It was decreased by 2.3% compared to the CPU result.",

keywords = "Bfloat16, Floating-point MAC unit, Long Short-Term Memory, Recurrent Neural Networks",

author = "Kim, {Seok Young} and Kim, {Chang Hyun} and Kim, {Seon Wook}",

note = "Funding Information: This work was supported in part by SK Hynix Inc. Publisher Copyright: {\textcopyright} 2021 IEEE.; 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021 ; Conference date: 27-06-2021 Through 30-06-2021",

year = "2021",

month = jun,

day = "27",

doi = "10.1109/ITC-CSCC52171.2021.9553004",

language = "English",

series = "2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021",

}

TY - GEN

T1 - Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells

AU - Kim, Seok Young

AU - Kim, Chang Hyun

AU - Kim, Seon Wook

PY - 2021/6/27

Y1 - 2021/6/27

N2 - The characteristics of recurrent and functional reduction in RNNs (Recurrent Neural Networks) may cause an inefficient execution in conventional hardware such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit). Their large number of recurrent data prevents the CPU from exploiting their locality, while the reduction feature limits the GPU parallelism. As a result, several dedicated hardware for efficient RNN execution has been proposed recently. In this paper, we propose a processing element that performs primary operations (i.e., multiplication and accumulation operation and activation function operation) of LSTM (Long Short-Term Memory), the most dominant model of RNN We implemented the processing element on an FPGA (Field-Programmable Gate Array) and verified the accuracy by measuring the BLEU score (Bilingual Evaluation Understudy score) of the PyTorch-based LSTM application. It was decreased by 2.3% compared to the CPU result.

AB - The characteristics of recurrent and functional reduction in RNNs (Recurrent Neural Networks) may cause an inefficient execution in conventional hardware such as CPU (Central Processing Unit) and GPU (Graphics Processing Unit). Their large number of recurrent data prevents the CPU from exploiting their locality, while the reduction feature limits the GPU parallelism. As a result, several dedicated hardware for efficient RNN execution has been proposed recently. In this paper, we propose a processing element that performs primary operations (i.e., multiplication and accumulation operation and activation function operation) of LSTM (Long Short-Term Memory), the most dominant model of RNN We implemented the processing element on an FPGA (Field-Programmable Gate Array) and verified the accuracy by measuring the BLEU score (Bilingual Evaluation Understudy score) of the PyTorch-based LSTM application. It was decreased by 2.3% compared to the CPU result.

KW - Bfloat16

KW - Floating-point MAC unit

KW - Long Short-Term Memory

KW - Recurrent Neural Networks

UR - http://www.scopus.com/inward/record.url?scp=85117080171&partnerID=8YFLogxK

U2 - 10.1109/ITC-CSCC52171.2021.9553004

DO - 10.1109/ITC-CSCC52171.2021.9553004

M3 - Conference contribution

AN - SCOPUS:85117080171

T3 - 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

BT - 2021 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 36th International Technical Conference on Circuits/Systems, Computers and Communications, ITC-CSCC 2021

Y2 - 27 June 2021 through 30 June 2021

ER -

Implementation of Pipelined Adder Tree for Long Short-Term Memory Cells

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this