Performance prediction for convolutional neural network on spark cluster

Rohyoung Myung, Heonchang Yu

Research output: Contribution to journalArticlepeer-review

6 Citations (Scopus)


Applications with large-scale data are processed on a distributed system, such as Spark, as they are data-and computation-intensive. Predicting the performance of such applications is difficult, because they are influenced by various aspects of configurations from the distributed framework level to the application level. In this paper, we propose a completion time prediction model based on machine learning for the representative deep learning model convolutional neural network (CNN) by analyzing the effects of data, task, and resource characteristics on performance when executing the model in Spark cluster. To reduce the time utilized in collecting the data for training the model, we consider the causal relationship between the model features and the completion time based on Spark CNN’s distributed data-parallel model. The model features include the configurations of the Data Center OS Mesos environment, configurations of Apache Spark, and configurations of the CNN model. By applying the proposed model to famous CNN implementations, we achieved 99.98% prediction accuracy about estimating the job completion time. In addition to the downscale search area for the model features, we leverage extrapolation, which significantly reduces the model build time at most to 89% with even better prediction accuracy in comparison to the actual work.

Original languageEnglish
Article number1340
Pages (from-to)1-22
Number of pages22
JournalElectronics (Switzerland)
Issue number9
Publication statusPublished - 2020 Sept

Bibliographical note

Funding Information:
Funding: This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the ITRC(Information Technology Research Center) support program (IITP-2018-0-01405) supervised by the IITP(Institute for Information & communications Technology Planning & Evaluation). This work was supported by Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (No.2018-0-00480, Developing the edge cloud platform for the real-time services based on the mobility of connected cars).

Publisher Copyright:
© 2020 by the authors. Licensee MDPI, Basel, Switzerland.


  • Convolutional neural network
  • Feature engineering
  • Machine learning
  • Performance prediction

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Signal Processing
  • Hardware and Architecture
  • Computer Networks and Communications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Performance prediction for convolutional neural network on spark cluster'. Together they form a unique fingerprint.

Cite this