Analysis of parallel training algorithms for deep neural networks

Hyowon Lee; Keonnyeong Lee; In Chul Yoo; Dongsuk Yook

doi:10.1109/CSCI46756.2018.00291

Analysis of parallel training algorithms for deep neural networks

Hyowon Lee, Keonnyeong Lee, In Chul Yoo, Dongsuk Yook

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

2 Citations (Scopus)

Abstract

A majority of training algorithms for deep neural networks (DNNs) use variants of stochastic gradient descent (SGD). Since a large amount of time is typically required to train DNNs, many attempts have been made to speed-up the training process by parallelizing the SGD algorithms. However, such parallelization efforts introduce approximation due to the inherent sequential nature of the SGD methods. In this paper, we revisit and analyze parallel SGD algorithms, and propose a novel pipelined SGD that is more efficient than previous algorithms.

Original language	English
Title of host publication	Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1462-1463
Number of pages	2
ISBN (Electronic)	9781728113609
DOIs	https://doi.org/10.1109/CSCI46756.2018.00291
Publication status	Published - 2018 Dec
Event	2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018 - Las Vegas, United States Duration: 2018 Dec 13 → 2018 Dec 15

Publication series

Name	Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018

Conference

Conference	2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018
Country/Territory	United States
City	Las Vegas
Period	18/12/13 → 18/12/15

Bibliographical note

Funding Information:
ACKNOWLEDGMENT This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (NRF-2017R1E1A1A01078157). Also, it was partly supported by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korean government (MSIP) (No.2018-0-00269, A research on safe and convenient big data processing methods).

Publisher Copyright:
© 2018 IEEE.

Keywords

Deep neural networks
Pipelined stochastic gradient descent
Stochastic gradient descent

ASJC Scopus subject areas

Computer Networks and Communications
Computer Science Applications
Hardware and Architecture
Information Systems and Management
Control and Optimization
Modelling and Simulation
Artificial Intelligence

Access to Document

10.1109/CSCI46756.2018.00291

Cite this

Lee, H., Lee, K., Yoo, I. C., & Yook, D. (2018). Analysis of parallel training algorithms for deep neural networks. In Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018 (pp. 1462-1463). Article 8947467 (Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CSCI46756.2018.00291

Analysis of parallel training algorithms for deep neural networks. / Lee, Hyowon; Lee, Keonnyeong; Yoo, In Chul et al.
Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1462-1463 8947467 (Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lee, H, Lee, K, Yoo, IC & Yook, D 2018, Analysis of parallel training algorithms for deep neural networks. in Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018., 8947467, Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018, Institute of Electrical and Electronics Engineers Inc., pp. 1462-1463, 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018, Las Vegas, United States, 18/12/13. https://doi.org/10.1109/CSCI46756.2018.00291

Lee H, Lee K, Yoo IC, Yook D. Analysis of parallel training algorithms for deep neural networks. In Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1462-1463. 8947467. (Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018). doi: 10.1109/CSCI46756.2018.00291

Lee, Hyowon ; Lee, Keonnyeong ; Yoo, In Chul et al. / Analysis of parallel training algorithms for deep neural networks. Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1462-1463 (Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018).

@inproceedings{deea61ab52824f478dd91cafb4ddeba6,

title = "Analysis of parallel training algorithms for deep neural networks",

abstract = "A majority of training algorithms for deep neural networks (DNNs) use variants of stochastic gradient descent (SGD). Since a large amount of time is typically required to train DNNs, many attempts have been made to speed-up the training process by parallelizing the SGD algorithms. However, such parallelization efforts introduce approximation due to the inherent sequential nature of the SGD methods. In this paper, we revisit and analyze parallel SGD algorithms, and propose a novel pipelined SGD that is more efficient than previous algorithms.",

keywords = "Deep neural networks, Pipelined stochastic gradient descent, Stochastic gradient descent",

author = "Hyowon Lee and Keonnyeong Lee and Yoo, {In Chul} and Dongsuk Yook",

note = "Funding Information: ACKNOWLEDGMENT This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (NRF-2017R1E1A1A01078157). Also, it was partly supported by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korean government (MSIP) (No.2018-0-00269, A research on safe and convenient big data processing methods). Publisher Copyright: {\textcopyright} 2018 IEEE.; 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018 ; Conference date: 13-12-2018 Through 15-12-2018",

year = "2018",

month = dec,

doi = "10.1109/CSCI46756.2018.00291",

language = "English",

series = "Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1462--1463",

booktitle = "Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018",

}

TY - GEN

T1 - Analysis of parallel training algorithms for deep neural networks

AU - Lee, Hyowon

AU - Lee, Keonnyeong

AU - Yoo, In Chul

AU - Yook, Dongsuk

N1 - Funding Information: ACKNOWLEDGMENT This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (NRF-2017R1E1A1A01078157). Also, it was partly supported by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korean government (MSIP) (No.2018-0-00269, A research on safe and convenient big data processing methods). Publisher Copyright: © 2018 IEEE.

PY - 2018/12

Y1 - 2018/12

N2 - A majority of training algorithms for deep neural networks (DNNs) use variants of stochastic gradient descent (SGD). Since a large amount of time is typically required to train DNNs, many attempts have been made to speed-up the training process by parallelizing the SGD algorithms. However, such parallelization efforts introduce approximation due to the inherent sequential nature of the SGD methods. In this paper, we revisit and analyze parallel SGD algorithms, and propose a novel pipelined SGD that is more efficient than previous algorithms.

AB - A majority of training algorithms for deep neural networks (DNNs) use variants of stochastic gradient descent (SGD). Since a large amount of time is typically required to train DNNs, many attempts have been made to speed-up the training process by parallelizing the SGD algorithms. However, such parallelization efforts introduce approximation due to the inherent sequential nature of the SGD methods. In this paper, we revisit and analyze parallel SGD algorithms, and propose a novel pipelined SGD that is more efficient than previous algorithms.

KW - Deep neural networks

KW - Pipelined stochastic gradient descent

KW - Stochastic gradient descent

UR - http://www.scopus.com/inward/record.url?scp=85078565376&partnerID=8YFLogxK

U2 - 10.1109/CSCI46756.2018.00291

DO - 10.1109/CSCI46756.2018.00291

M3 - Conference contribution

AN - SCOPUS:85078565376

T3 - Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018

SP - 1462

EP - 1463

BT - Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018

Y2 - 13 December 2018 through 15 December 2018

ER -

Analysis of parallel training algorithms for deep neural networks

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this