Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead

Hochan Lee; Heewon Kim; Chanbin Bae; Yujin Kim; Sangheon Pack

doi:10.1145/3624354.3630090

Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead

Hochan Lee, Heewon Kim, Chanbin Bae, Yujin Kim, Sangheon Pack

School of Electrical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

1 Citation (Scopus)

Abstract

In-network aggregation (INA) can accelerate distributed deep learning (DDL) by offloading a gradient aggregation operation to programmable switches. However, as the number of jobs increases, the communication overhead increases since the INA switch's resources for aggregation are limited. To reduce the communication overhead, gradient quantization approaches that compress gradients into fewer bits can be used. Although these approaches are effective in INA, existing INA schemes cannot efficiently use the switch's resources for low-bit quantized gradients. In this paper, we propose a quantization-aware in-network aggregation (QAINA) scheme that provides flexible aggregation and achieves efficient resource utilization for low-bit quantized gradients.

Original language	English
Title of host publication	CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies
Publisher	Association for Computing Machinery, Inc
Pages	57-58
Number of pages	2
ISBN (Electronic)	9798400704079
DOIs	https://doi.org/10.1145/3624354.3630090
Publication status	Published - 2023 Dec 5
Event	19th International Conference on emerging Networking EXperiments and Technologies, CoNEXT Companion 2023 - Paris, France Duration: 2023 Dec 5 → 2023 Dec 8

Publication series

Name	CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies

Conference

Conference	19th International Conference on emerging Networking EXperiments and Technologies, CoNEXT Companion 2023
Country/Territory	France
City	Paris
Period	23/12/5 → 23/12/8

Bibliographical note

Publisher Copyright:
© 2023 Owner/Author.

Keywords

gradient quantization
in-network aggregation

ASJC Scopus subject areas

Computer Networks and Communications
Computer Science Applications
Hardware and Architecture

Access to Document

10.1145/3624354.3630090

Cite this

Lee, H., Kim, H., Bae, C., Kim, Y., & Pack, S. (2023). Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead. In CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies (pp. 57-58). (CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies). Association for Computing Machinery, Inc. https://doi.org/10.1145/3624354.3630090

Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead. / Lee, Hochan; Kim, Heewon; Bae, Chanbin et al.
CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies. Association for Computing Machinery, Inc, 2023. p. 57-58 (CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Lee, H, Kim, H, Bae, C, Kim, Y & Pack, S 2023, Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead. in CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies. CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies, Association for Computing Machinery, Inc, pp. 57-58, 19th International Conference on emerging Networking EXperiments and Technologies, CoNEXT Companion 2023, Paris, France, 23/12/5. https://doi.org/10.1145/3624354.3630090

Lee H, Kim H, Bae C, Kim Y, Pack S. Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead. In CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies. Association for Computing Machinery, Inc. 2023. p. 57-58. (CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies). doi: 10.1145/3624354.3630090

Lee, Hochan ; Kim, Heewon ; Bae, Chanbin et al. / Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead. CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies. Association for Computing Machinery, Inc, 2023. pp. 57-58 (CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies).

@inproceedings{a50542f0f64c4478b79f5a76647bb74f,

title = "Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead",

abstract = "In-network aggregation (INA) can accelerate distributed deep learning (DDL) by offloading a gradient aggregation operation to programmable switches. However, as the number of jobs increases, the communication overhead increases since the INA switch's resources for aggregation are limited. To reduce the communication overhead, gradient quantization approaches that compress gradients into fewer bits can be used. Although these approaches are effective in INA, existing INA schemes cannot efficiently use the switch's resources for low-bit quantized gradients. In this paper, we propose a quantization-aware in-network aggregation (QAINA) scheme that provides flexible aggregation and achieves efficient resource utilization for low-bit quantized gradients.",

keywords = "gradient quantization, in-network aggregation",

author = "Hochan Lee and Heewon Kim and Chanbin Bae and Yujin Kim and Sangheon Pack",

note = "Publisher Copyright: {\textcopyright} 2023 Owner/Author.; 19th International Conference on emerging Networking EXperiments and Technologies, CoNEXT Companion 2023 ; Conference date: 05-12-2023 Through 08-12-2023",

year = "2023",

month = dec,

day = "5",

doi = "10.1145/3624354.3630090",

language = "English",

series = "CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies",

publisher = "Association for Computing Machinery, Inc",

pages = "57--58",

booktitle = "CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies",

}

TY - GEN

T1 - Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead

AU - Lee, Hochan

AU - Kim, Heewon

AU - Bae, Chanbin

AU - Kim, Yujin

AU - Pack, Sangheon

PY - 2023/12/5

Y1 - 2023/12/5

N2 - In-network aggregation (INA) can accelerate distributed deep learning (DDL) by offloading a gradient aggregation operation to programmable switches. However, as the number of jobs increases, the communication overhead increases since the INA switch's resources for aggregation are limited. To reduce the communication overhead, gradient quantization approaches that compress gradients into fewer bits can be used. Although these approaches are effective in INA, existing INA schemes cannot efficiently use the switch's resources for low-bit quantized gradients. In this paper, we propose a quantization-aware in-network aggregation (QAINA) scheme that provides flexible aggregation and achieves efficient resource utilization for low-bit quantized gradients.

AB - In-network aggregation (INA) can accelerate distributed deep learning (DDL) by offloading a gradient aggregation operation to programmable switches. However, as the number of jobs increases, the communication overhead increases since the INA switch's resources for aggregation are limited. To reduce the communication overhead, gradient quantization approaches that compress gradients into fewer bits can be used. Although these approaches are effective in INA, existing INA schemes cannot efficiently use the switch's resources for low-bit quantized gradients. In this paper, we propose a quantization-aware in-network aggregation (QAINA) scheme that provides flexible aggregation and achieves efficient resource utilization for low-bit quantized gradients.

KW - gradient quantization

KW - in-network aggregation

UR - http://www.scopus.com/inward/record.url?scp=85183584741&partnerID=8YFLogxK

U2 - 10.1145/3624354.3630090

DO - 10.1145/3624354.3630090

M3 - Conference contribution

AN - SCOPUS:85183584741

T3 - CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies

SP - 57

EP - 58

BT - CoNEXT Companion 2023 - Companion of the 19th International Conference on emerging Networking EXperiments and Technologies

PB - Association for Computing Machinery, Inc

T2 - 19th International Conference on emerging Networking EXperiments and Technologies, CoNEXT Companion 2023

Y2 - 5 December 2023 through 8 December 2023

ER -

Quantization-Aware In-Network Aggregation for Minimizing Communication Overhead

Abstract

Publication series

Conference

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this