Prediction of the Resource Consumption of Distributed Deep Learning Systems

Gyeongsik Yang; Changyong Shin; Jeunghwan Lee; Yeonho Yoo; Chuck Yoo

doi:10.1145/3489048.3530962

Prediction of the Resource Consumption of Distributed Deep Learning Systems

Gyeongsik Yang, Changyong Shin, Jeunghwan Lee, Yeonho Yoo, Chuck Yoo

Department of Computer Science and Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

Predicting resource consumption for the distributed training of deep learning models is of paramount importance, as it can inform a priori users of how long their training would take and enable users to manage the cost of training. Yet, no such prediction is available for users because the resource consumption itself varies significantly according to "settings"such as GPU types and also by "workloads"like deep learning models. Previous studies have attempted to derive or model such a prediction, but they fall short of accommodating the various combinations of settings and workloads together. This study presents Driple, which designs graph neural networks to predict the resource consumption of diverse workloads. Driple also designs transfer learning to extend the graph neural networks to adapt to differences in settings. The evaluation results show that Driple effectively predicts a wide range of workloads and settings. In addition, Driple can efficiently reduce the time required to tailor the prediction for different settings by up to 7.3×.

Original language	English
Title of host publication	SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems
Publisher	Association for Computing Machinery, Inc
Pages	69-70
Number of pages	2
ISBN (Electronic)	9781450391412
DOIs	https://doi.org/10.1145/3489048.3530962
Publication status	Published - 2022 Jun 6
Event	2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS/PERFORMANCE 2022 - Virtual, Online, India Duration: 2022 Jun 6 → 2022 Jun 10

Publication series

Name	SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems

Conference

Conference	2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS/PERFORMANCE 2022
Country/Territory	India
City	Virtual, Online
Period	22/6/6 → 22/6/10

Bibliographical note

Funding Information:
This work was supported by Institute of Information & communications Technology Planning & Evaluation grant funded by the Korea government (Ministry of Science and ICT) (2015-0-00280) and by Basic Science Research Program through the NRF funded by the Ministry of Education (NRF-2021R1A6A1A13044830).

Publisher Copyright:
© 2022 Owner/Author.

Keywords

distributed deep learning
graph neural networks
resource prediction
training time prediction
transfer learning

ASJC Scopus subject areas

Computational Theory and Mathematics
Computer Networks and Communications
Hardware and Architecture
Software

Access to Document

10.1145/3489048.3530962

Cite this

Yang, G., Shin, C., Lee, J., Yoo, Y., & Yoo, C. (2022). Prediction of the Resource Consumption of Distributed Deep Learning Systems. In SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems (pp. 69-70). (SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems). Association for Computing Machinery, Inc. https://doi.org/10.1145/3489048.3530962

Prediction of the Resource Consumption of Distributed Deep Learning Systems. / Yang, Gyeongsik; Shin, Changyong; Lee, Jeunghwan et al.
SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems. Association for Computing Machinery, Inc, 2022. p. 69-70 (SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yang, G, Shin, C, Lee, J, Yoo, Y & Yoo, C 2022, Prediction of the Resource Consumption of Distributed Deep Learning Systems. in SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems. SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, Association for Computing Machinery, Inc, pp. 69-70, 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS/PERFORMANCE 2022, Virtual, Online, India, 22/6/6. https://doi.org/10.1145/3489048.3530962

Yang G, Shin C, Lee J, Yoo Y, Yoo C. Prediction of the Resource Consumption of Distributed Deep Learning Systems. In SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems. Association for Computing Machinery, Inc. 2022. p. 69-70. (SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems). doi: 10.1145/3489048.3530962

Yang, Gyeongsik ; Shin, Changyong ; Lee, Jeunghwan et al. / Prediction of the Resource Consumption of Distributed Deep Learning Systems. SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems. Association for Computing Machinery, Inc, 2022. pp. 69-70 (SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems).

@inproceedings{608f925fe49244ef802d8d6373fcbe0e,

title = "Prediction of the Resource Consumption of Distributed Deep Learning Systems",

abstract = "Predicting resource consumption for the distributed training of deep learning models is of paramount importance, as it can inform a priori users of how long their training would take and enable users to manage the cost of training. Yet, no such prediction is available for users because the resource consumption itself varies significantly according to {"}settings{"}such as GPU types and also by {"}workloads{"}like deep learning models. Previous studies have attempted to derive or model such a prediction, but they fall short of accommodating the various combinations of settings and workloads together. This study presents Driple, which designs graph neural networks to predict the resource consumption of diverse workloads. Driple also designs transfer learning to extend the graph neural networks to adapt to differences in settings. The evaluation results show that Driple effectively predicts a wide range of workloads and settings. In addition, Driple can efficiently reduce the time required to tailor the prediction for different settings by up to 7.3×.",

keywords = "distributed deep learning, graph neural networks, resource prediction, training time prediction, transfer learning",

author = "Gyeongsik Yang and Changyong Shin and Jeunghwan Lee and Yeonho Yoo and Chuck Yoo",

note = "Funding Information: This work was supported by Institute of Information & communications Technology Planning & Evaluation grant funded by the Korea government (Ministry of Science and ICT) (2015-0-00280) and by Basic Science Research Program through the NRF funded by the Ministry of Education (NRF-2021R1A6A1A13044830). Publisher Copyright: {\textcopyright} 2022 Owner/Author.; 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS/PERFORMANCE 2022 ; Conference date: 06-06-2022 Through 10-06-2022",

year = "2022",

month = jun,

day = "6",

doi = "10.1145/3489048.3530962",

language = "English",

series = "SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems",

publisher = "Association for Computing Machinery, Inc",

pages = "69--70",

booktitle = "SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems",

}

TY - GEN

T1 - Prediction of the Resource Consumption of Distributed Deep Learning Systems

AU - Yang, Gyeongsik

AU - Shin, Changyong

AU - Lee, Jeunghwan

AU - Yoo, Yeonho

AU - Yoo, Chuck

N1 - Funding Information: This work was supported by Institute of Information & communications Technology Planning & Evaluation grant funded by the Korea government (Ministry of Science and ICT) (2015-0-00280) and by Basic Science Research Program through the NRF funded by the Ministry of Education (NRF-2021R1A6A1A13044830). Publisher Copyright: © 2022 Owner/Author.

PY - 2022/6/6

Y1 - 2022/6/6

N2 - Predicting resource consumption for the distributed training of deep learning models is of paramount importance, as it can inform a priori users of how long their training would take and enable users to manage the cost of training. Yet, no such prediction is available for users because the resource consumption itself varies significantly according to "settings"such as GPU types and also by "workloads"like deep learning models. Previous studies have attempted to derive or model such a prediction, but they fall short of accommodating the various combinations of settings and workloads together. This study presents Driple, which designs graph neural networks to predict the resource consumption of diverse workloads. Driple also designs transfer learning to extend the graph neural networks to adapt to differences in settings. The evaluation results show that Driple effectively predicts a wide range of workloads and settings. In addition, Driple can efficiently reduce the time required to tailor the prediction for different settings by up to 7.3×.

AB - Predicting resource consumption for the distributed training of deep learning models is of paramount importance, as it can inform a priori users of how long their training would take and enable users to manage the cost of training. Yet, no such prediction is available for users because the resource consumption itself varies significantly according to "settings"such as GPU types and also by "workloads"like deep learning models. Previous studies have attempted to derive or model such a prediction, but they fall short of accommodating the various combinations of settings and workloads together. This study presents Driple, which designs graph neural networks to predict the resource consumption of diverse workloads. Driple also designs transfer learning to extend the graph neural networks to adapt to differences in settings. The evaluation results show that Driple effectively predicts a wide range of workloads and settings. In addition, Driple can efficiently reduce the time required to tailor the prediction for different settings by up to 7.3×.

KW - distributed deep learning

KW - graph neural networks

KW - resource prediction

KW - training time prediction

KW - transfer learning

UR - http://www.scopus.com/inward/record.url?scp=85132196416&partnerID=8YFLogxK

U2 - 10.1145/3489048.3530962

DO - 10.1145/3489048.3530962

M3 - Conference contribution

AN - SCOPUS:85132196416

T3 - SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems

SP - 69

EP - 70

BT - SIGMETRICS/PERFORMANCE 2022 - Abstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems

PB - Association for Computing Machinery, Inc

T2 - 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS/PERFORMANCE 2022

Y2 - 6 June 2022 through 10 June 2022

ER -

Prediction of the Resource Consumption of Distributed Deep Learning Systems

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this