TensorExpress: In-network communication scheduling for distributed deep learning

Minkoo Kang; Gyeongsik Yang; Yeonho Yoo; Chuck Yoo

doi:10.1109/CLOUD49709.2020.00014

TensorExpress: In-network communication scheduling for distributed deep learning

Minkoo Kang
, Gyeongsik Yang
, Yeonho Yoo
, Chuck Yoo^*

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

15 Citations (Scopus)

Abstract

TensorExpress provides in-network communication scheduling for distributed deep learning (DDL). In cloud-based DDL, parameter communication over a network is a key bottleneck. Previous studies proposed tensor packet reordering approaches to reduce network blocking time. However, network contention still exists in DDL. TensorExpress mitigates network contention and reduces overall training time. It schedules tensor packets in-network using P4, a switch programming language. TensorExpress improves latency and network blocking time up to 2.5 and 2.44 times, respectively.

Original language	English
Title of host publication	Proceedings - 2020 IEEE 13th International Conference on Cloud Computing, CLOUD 2020
Publisher	IEEE Computer Society
Pages	25-27
Number of pages	3
ISBN (Electronic)	9781728187808
DOIs	https://doi.org/10.1109/CLOUD49709.2020.00014
Publication status	Published - 2020 Oct
Event	13th IEEE International Conference on Cloud Computing, CLOUD 2020 - Virtual, Beijing, China Duration: 2020 Oct 18 → 2020 Oct 24

Publication series

Name	IEEE International Conference on Cloud Computing, CLOUD
Volume	2020-October
ISSN (Print)	2159-6182
ISSN (Electronic)	2159-6190

Conference

Conference	13th IEEE International Conference on Cloud Computing, CLOUD 2020
Country/Territory	China
City	Virtual, Beijing
Period	20/10/18 → 20/10/24

Bibliographical note

Funding Information:
†This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation grant funded by the Korea government (No. 2015-0-00280, (SW Starlab) Next generation cloud infra-software toward the guarantee of performance and security SLA). This research was also supported by National Research Foundation of Korea funded by the Ministry of Science, ICT (No. NRF-2019H1D8A2105513).

Publisher Copyright:
© 2020 IEEE.

Keywords

Communication scheduling
Distributed deep learning
In-network delay
P4
Parameter server architecture

ASJC Scopus subject areas

Artificial Intelligence
Information Systems
Software

Access to Document

10.1109/CLOUD49709.2020.00014

Cite this

Kang, M., Yang, G., Yoo, Y., & Yoo, C. (2020). TensorExpress: In-network communication scheduling for distributed deep learning. In Proceedings - 2020 IEEE 13th International Conference on Cloud Computing, CLOUD 2020 (pp. 25-27). Article 9284207 (IEEE International Conference on Cloud Computing, CLOUD; Vol. 2020-October). IEEE Computer Society. https://doi.org/10.1109/CLOUD49709.2020.00014

TensorExpress: In-network communication scheduling for distributed deep learning. / Kang, Minkoo; Yang, Gyeongsik; Yoo, Yeonho et al.
Proceedings - 2020 IEEE 13th International Conference on Cloud Computing, CLOUD 2020. IEEE Computer Society, 2020. p. 25-27 9284207 (IEEE International Conference on Cloud Computing, CLOUD; Vol. 2020-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Kang, M, Yang, G, Yoo, Y & Yoo, C 2020, TensorExpress: In-network communication scheduling for distributed deep learning. in Proceedings - 2020 IEEE 13th International Conference on Cloud Computing, CLOUD 2020., 9284207, IEEE International Conference on Cloud Computing, CLOUD, vol. 2020-October, IEEE Computer Society, pp. 25-27, 13th IEEE International Conference on Cloud Computing, CLOUD 2020, Virtual, Beijing, China, 20/10/18. https://doi.org/10.1109/CLOUD49709.2020.00014

@inproceedings{36750fed26e14ab68f27e1629dcb0aab,

title = "TensorExpress: In-network communication scheduling for distributed deep learning",

abstract = "TensorExpress provides in-network communication scheduling for distributed deep learning (DDL). In cloud-based DDL, parameter communication over a network is a key bottleneck. Previous studies proposed tensor packet reordering approaches to reduce network blocking time. However, network contention still exists in DDL. TensorExpress mitigates network contention and reduces overall training time. It schedules tensor packets in-network using P4, a switch programming language. TensorExpress improves latency and network blocking time up to 2.5 and 2.44 times, respectively.",

keywords = "Communication scheduling, Distributed deep learning, In-network delay, P4, Parameter server architecture",

author = "Minkoo Kang and Gyeongsik Yang and Yeonho Yoo and Chuck Yoo",

note = "Funding Information: †This work was partly supported by Institute of Information \& Communications Technology Planning \& Evaluation grant funded by the Korea government (No. 2015-0-00280, (SW Starlab) Next generation cloud infra-software toward the guarantee of performance and security SLA). This research was also supported by National Research Foundation of Korea funded by the Ministry of Science, ICT (No. NRF-2019H1D8A2105513). Publisher Copyright: {\textcopyright} 2020 IEEE.; 13th IEEE International Conference on Cloud Computing, CLOUD 2020 ; Conference date: 18-10-2020 Through 24-10-2020",

year = "2020",

month = oct,

doi = "10.1109/CLOUD49709.2020.00014",

language = "English",

series = "IEEE International Conference on Cloud Computing, CLOUD",

publisher = "IEEE Computer Society",

pages = "25--27",

booktitle = "Proceedings - 2020 IEEE 13th International Conference on Cloud Computing, CLOUD 2020",

}

TY - GEN

T1 - TensorExpress

T2 - 13th IEEE International Conference on Cloud Computing, CLOUD 2020

AU - Kang, Minkoo

AU - Yang, Gyeongsik

AU - Yoo, Yeonho

AU - Yoo, Chuck

N1 - Funding Information: †This work was partly supported by Institute of Information & Communications Technology Planning & Evaluation grant funded by the Korea government (No. 2015-0-00280, (SW Starlab) Next generation cloud infra-software toward the guarantee of performance and security SLA). This research was also supported by National Research Foundation of Korea funded by the Ministry of Science, ICT (No. NRF-2019H1D8A2105513). Publisher Copyright: © 2020 IEEE.

PY - 2020/10

Y1 - 2020/10

N2 - TensorExpress provides in-network communication scheduling for distributed deep learning (DDL). In cloud-based DDL, parameter communication over a network is a key bottleneck. Previous studies proposed tensor packet reordering approaches to reduce network blocking time. However, network contention still exists in DDL. TensorExpress mitigates network contention and reduces overall training time. It schedules tensor packets in-network using P4, a switch programming language. TensorExpress improves latency and network blocking time up to 2.5 and 2.44 times, respectively.

AB - TensorExpress provides in-network communication scheduling for distributed deep learning (DDL). In cloud-based DDL, parameter communication over a network is a key bottleneck. Previous studies proposed tensor packet reordering approaches to reduce network blocking time. However, network contention still exists in DDL. TensorExpress mitigates network contention and reduces overall training time. It schedules tensor packets in-network using P4, a switch programming language. TensorExpress improves latency and network blocking time up to 2.5 and 2.44 times, respectively.

KW - Communication scheduling

KW - Distributed deep learning

KW - In-network delay

KW - P4

KW - Parameter server architecture

UR - https://www.scopus.com/pages/publications/85098592953

U2 - 10.1109/CLOUD49709.2020.00014

DO - 10.1109/CLOUD49709.2020.00014

M3 - Conference contribution

AN - SCOPUS:85098592953

T3 - IEEE International Conference on Cloud Computing, CLOUD

SP - 25

EP - 27

BT - Proceedings - 2020 IEEE 13th International Conference on Cloud Computing, CLOUD 2020

PB - IEEE Computer Society

Y2 - 18 October 2020 through 24 October 2020

ER -

TensorExpress: In-network communication scheduling for distributed deep learning

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this