Pipeline Parallelism With Elastic Averaging

Bongwon Jang, In Chul Yoo, Dongsuk Yook

Research output: Contribution to journalArticlepeer-review

Abstract

To accelerate the training speed of massive DNN models on large-scale datasets, distributed training techniques, including data parallelism and model parallelism, have been extensively studied. In particular, pipeline parallelism, which is derived from model parallelism, has been attracting attention. It splits the model parameters across multiple computing nodes and executes multiple mini-batches simultaneously. However, naive pipeline parallelism suffers from the issues of weight inconsistency and delayed gradients, as the model parameters used in the forward and backward passes do not match, causing unstable training and low performance. In this study, we propose a novel pipeline parallelism technique called EA-Pipe to address the weight inconsistency and delayed gradient problems. EA-Pipe applies an elastic averaging method, which has been studied in the context of data parallelism, to pipeline parallelism. The proposed method maintains multiple model replicas to solve the weight inconsistency problem, and synchronizes the model replicas using an elasticity-based moving average method to mitigate the delayed gradient problem. To verify the efficacy of the proposed method, we conducted three image classification experiments on the CIFAR-10/100 and ImageNet datasets. The experimental results show that EA-Pipe not only accelerates training speed but also demonstrates more stable learning property compared to existing pipeline parallelism techniques. Especially, in the experiments using the CIFAR-100 and ImageNet datasets, EA-Pipe recorded error rates that were 2.58% and 2.19% lower, respectively, than the baseline pipeline parallelization method.

Original languageEnglish
Pages (from-to)5477-5489
Number of pages13
JournalIEEE Access
Volume12
DOIs
Publication statusPublished - 2024

Bibliographical note

Publisher Copyright:
© 2013 IEEE.

Keywords

  • Deep learning
  • parallel processing
  • pipeline processing
  • stochastic gradient descent (SGD)

ASJC Scopus subject areas

  • General Computer Science
  • General Materials Science
  • General Engineering

Fingerprint

Dive into the research topics of 'Pipeline Parallelism With Elastic Averaging'. Together they form a unique fingerprint.

Cite this