Learning to Balance Local Losses via Meta-Learning

Seungdong Yoa; Minkyu Jeon; Youngjin Oh; Hyunwoo J. Kim

doi:10.1109/ACCESS.2021.3113934

Learning to Balance Local Losses via Meta-Learning

Seungdong Yoa, Minkyu Jeon, Youngjin Oh, Hyunwoo J. Kim

Research output: Contribution to journal › Article › peer-review

1 Citation (Scopus)

Abstract

The standard training for deep neural networks relies on a global and fixed loss function. For more effective training, dynamic loss functions have been recently proposed. However, the dynamic global loss function is not flexible to differentially train layers in complex deep neural networks. In this paper, we propose a general framework that learns to adaptively train each layer of deep neural networks via meta-learning. Our framework leverages the local error signals from layers and identifies which layer needs to be trained more at every iteration. Also, the proposed method improves the local loss function with our minibatch-wise dropout and cross-validation loop to alleviate meta-overfitting. The experiments show that our method achieved competitive performance compared to state-of-the-art methods on popular benchmark datasets for image classification: CIFAR-10 and CIFAR-100. Surprisingly, our method enables training deep neural networks without skip-connections using dynamically weighted local loss functions.

Original language	English
Pages (from-to)	130834-130844
Number of pages	11
Journal	IEEE Access
Volume	9
DOIs	https://doi.org/10.1109/ACCESS.2021.3113934
Publication status	Published - 2021

Bibliographical note

Funding Information:
This work was supported in part by the Institute for Information and Communications Technology Planning and Evaluation (IITP) Grant by Korean Government through the Ministry of Science and ICT (MSIT) (Regional strategic industry convergence security core talent training business) under Grant 2019-0-01343, in part by the ICT Creative Consilience Program supervised by IITP under Grant IITP-2021-2020-0-01819, in part by the Electronics and Telecommunications Research Institute (ETRI) Grant by Korean Government (Fundamental technology research for human-centric autonomous intelligent systems) under Grant 21ZS1200, and in part by Korea Institute of Planning and Evaluation for Technology in Food, Agriculture and Forestry (IPET) and Korea Smart Farm Research and Development Foundation (KosFarm) through the Smart Farm Innovation Technology Development Program by the Ministry of Agriculture, Food and Rural Affairs (MAFRA), and MSIT, Rural Development Administration (RDA), under Grant 421025-04.

Publisher Copyright:
© 2013 IEEE.

Keywords

Deep learning
image classification
machine learning
meta-learning

ASJC Scopus subject areas

General Computer Science
General Materials Science
General Engineering

Access to Document

10.1109/ACCESS.2021.3113934

Cite this

@article{f08368d774c84c79beeef19e24c91eb5,

title = "Learning to Balance Local Losses via Meta-Learning",

abstract = "The standard training for deep neural networks relies on a global and fixed loss function. For more effective training, dynamic loss functions have been recently proposed. However, the dynamic global loss function is not flexible to differentially train layers in complex deep neural networks. In this paper, we propose a general framework that learns to adaptively train each layer of deep neural networks via meta-learning. Our framework leverages the local error signals from layers and identifies which layer needs to be trained more at every iteration. Also, the proposed method improves the local loss function with our minibatch-wise dropout and cross-validation loop to alleviate meta-overfitting. The experiments show that our method achieved competitive performance compared to state-of-the-art methods on popular benchmark datasets for image classification: CIFAR-10 and CIFAR-100. Surprisingly, our method enables training deep neural networks without skip-connections using dynamically weighted local loss functions.",

keywords = "Deep learning, image classification, machine learning, meta-learning",

author = "Seungdong Yoa and Minkyu Jeon and Youngjin Oh and Kim, {Hyunwoo J.}",

note = "Funding Information: This work was supported in part by the Institute for Information and Communications Technology Planning and Evaluation (IITP) Grant by Korean Government through the Ministry of Science and ICT (MSIT) (Regional strategic industry convergence security core talent training business) under Grant 2019-0-01343, in part by the ICT Creative Consilience Program supervised by IITP under Grant IITP-2021-2020-0-01819, in part by the Electronics and Telecommunications Research Institute (ETRI) Grant by Korean Government (Fundamental technology research for human-centric autonomous intelligent systems) under Grant 21ZS1200, and in part by Korea Institute of Planning and Evaluation for Technology in Food, Agriculture and Forestry (IPET) and Korea Smart Farm Research and Development Foundation (KosFarm) through the Smart Farm Innovation Technology Development Program by the Ministry of Agriculture, Food and Rural Affairs (MAFRA), and MSIT, Rural Development Administration (RDA), under Grant 421025-04. Publisher Copyright: {\textcopyright} 2013 IEEE.",

year = "2021",

doi = "10.1109/ACCESS.2021.3113934",

language = "English",

volume = "9",

pages = "130834--130844",

journal = "IEEE Access",

issn = "2169-3536",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - JOUR

T1 - Learning to Balance Local Losses via Meta-Learning

AU - Yoa, Seungdong

AU - Jeon, Minkyu

AU - Oh, Youngjin

AU - Kim, Hyunwoo J.

N1 - Funding Information: This work was supported in part by the Institute for Information and Communications Technology Planning and Evaluation (IITP) Grant by Korean Government through the Ministry of Science and ICT (MSIT) (Regional strategic industry convergence security core talent training business) under Grant 2019-0-01343, in part by the ICT Creative Consilience Program supervised by IITP under Grant IITP-2021-2020-0-01819, in part by the Electronics and Telecommunications Research Institute (ETRI) Grant by Korean Government (Fundamental technology research for human-centric autonomous intelligent systems) under Grant 21ZS1200, and in part by Korea Institute of Planning and Evaluation for Technology in Food, Agriculture and Forestry (IPET) and Korea Smart Farm Research and Development Foundation (KosFarm) through the Smart Farm Innovation Technology Development Program by the Ministry of Agriculture, Food and Rural Affairs (MAFRA), and MSIT, Rural Development Administration (RDA), under Grant 421025-04. Publisher Copyright: © 2013 IEEE.

PY - 2021

Y1 - 2021

N2 - The standard training for deep neural networks relies on a global and fixed loss function. For more effective training, dynamic loss functions have been recently proposed. However, the dynamic global loss function is not flexible to differentially train layers in complex deep neural networks. In this paper, we propose a general framework that learns to adaptively train each layer of deep neural networks via meta-learning. Our framework leverages the local error signals from layers and identifies which layer needs to be trained more at every iteration. Also, the proposed method improves the local loss function with our minibatch-wise dropout and cross-validation loop to alleviate meta-overfitting. The experiments show that our method achieved competitive performance compared to state-of-the-art methods on popular benchmark datasets for image classification: CIFAR-10 and CIFAR-100. Surprisingly, our method enables training deep neural networks without skip-connections using dynamically weighted local loss functions.

AB - The standard training for deep neural networks relies on a global and fixed loss function. For more effective training, dynamic loss functions have been recently proposed. However, the dynamic global loss function is not flexible to differentially train layers in complex deep neural networks. In this paper, we propose a general framework that learns to adaptively train each layer of deep neural networks via meta-learning. Our framework leverages the local error signals from layers and identifies which layer needs to be trained more at every iteration. Also, the proposed method improves the local loss function with our minibatch-wise dropout and cross-validation loop to alleviate meta-overfitting. The experiments show that our method achieved competitive performance compared to state-of-the-art methods on popular benchmark datasets for image classification: CIFAR-10 and CIFAR-100. Surprisingly, our method enables training deep neural networks without skip-connections using dynamically weighted local loss functions.

KW - Deep learning

KW - image classification

KW - machine learning

KW - meta-learning

UR - http://www.scopus.com/inward/record.url?scp=85115668921&partnerID=8YFLogxK

U2 - 10.1109/ACCESS.2021.3113934

DO - 10.1109/ACCESS.2021.3113934

M3 - Article

AN - SCOPUS:85115668921

SN - 2169-3536

VL - 9

SP - 130834

EP - 130844

JO - IEEE Access

JF - IEEE Access

ER -

Learning to Balance Local Losses via Meta-Learning

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this