TY - GEN
T1 - Analysis of parallel training algorithms for deep neural networks
AU - Lee, Hyowon
AU - Lee, Keonnyeong
AU - Yoo, In Chul
AU - Yook, Dongsuk
N1 - Funding Information:
ACKNOWLEDGMENT This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (NRF-2017R1E1A1A01078157). Also, it was partly supported by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korean government (MSIP) (No.2018-0-00269, A research on safe and convenient big data processing methods).
Publisher Copyright:
© 2018 IEEE.
PY - 2018/12
Y1 - 2018/12
N2 - A majority of training algorithms for deep neural networks (DNNs) use variants of stochastic gradient descent (SGD). Since a large amount of time is typically required to train DNNs, many attempts have been made to speed-up the training process by parallelizing the SGD algorithms. However, such parallelization efforts introduce approximation due to the inherent sequential nature of the SGD methods. In this paper, we revisit and analyze parallel SGD algorithms, and propose a novel pipelined SGD that is more efficient than previous algorithms.
AB - A majority of training algorithms for deep neural networks (DNNs) use variants of stochastic gradient descent (SGD). Since a large amount of time is typically required to train DNNs, many attempts have been made to speed-up the training process by parallelizing the SGD algorithms. However, such parallelization efforts introduce approximation due to the inherent sequential nature of the SGD methods. In this paper, we revisit and analyze parallel SGD algorithms, and propose a novel pipelined SGD that is more efficient than previous algorithms.
KW - Deep neural networks
KW - Pipelined stochastic gradient descent
KW - Stochastic gradient descent
UR - http://www.scopus.com/inward/record.url?scp=85078565376&partnerID=8YFLogxK
U2 - 10.1109/CSCI46756.2018.00291
DO - 10.1109/CSCI46756.2018.00291
M3 - Conference contribution
AN - SCOPUS:85078565376
T3 - Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018
SP - 1462
EP - 1463
BT - Proceedings - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2018 International Conference on Computational Science and Computational Intelligence, CSCI 2018
Y2 - 13 December 2018 through 15 December 2018
ER -