Efficient backprop

Yann A. LeCun; Léon Bottou; Genevieve B. Orr; Klaus Robert Müller

doi:10.1007/978-3-642-35289-8_3

Efficient backprop

Yann A. LeCun, Léon Bottou, Genevieve B. Orr, Klaus Robert Müller

Research output: Chapter in Book/Report/Conference proceeding › Chapter

1664 Citations (Scopus)

Abstract

The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most "classical" second-order methods are impractical for large neural networks. A few methods are proposed that do not have these limitations.

Original language	English
Title of host publication	Neural Networks
Subtitle of host publication	Tricks of the Trade
Publisher	Springer Verlag
Pages	9-48
Number of pages	40
ISBN (Print)	9783642352881
DOIs	https://doi.org/10.1007/978-3-642-35289-8_3
Publication status	Published - 2012

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	7700 LECTURE NO
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-642-35289-8_3

Cite this

Efficient backprop. / LeCun, Yann A.; Bottou, Léon; Orr, Genevieve B. et al.
Neural Networks: Tricks of the Trade. Springer Verlag, 2012. p. 9-48 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 7700 LECTURE NO).

Research output: Chapter in Book/Report/Conference proceeding › Chapter

@inbook{b2b54acf0d394e59937cbf715d67502a,

title = "Efficient backprop",

abstract = "The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most {"}classical{"} second-order methods are impractical for large neural networks. A few methods are proposed that do not have these limitations.",

author = "LeCun, {Yann A.} and L{\'e}on Bottou and Orr, {Genevieve B.} and M{\"u}ller, {Klaus Robert}",

year = "2012",

doi = "10.1007/978-3-642-35289-8_3",

language = "English",

isbn = "9783642352881",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "9--48",

booktitle = "Neural Networks",

}

TY - CHAP

T1 - Efficient backprop

AU - LeCun, Yann A.

AU - Bottou, Léon

AU - Orr, Genevieve B.

AU - Müller, Klaus Robert

PY - 2012

Y1 - 2012

N2 - The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most "classical" second-order methods are impractical for large neural networks. A few methods are proposed that do not have these limitations.

AB - The convergence of back-propagation learning is analyzed so as to explain common phenomenon observed by practitioners. Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications. This paper gives some of those tricks, and offers explanations of why they work. Many authors have suggested that second-order optimization methods are advantageous for neural net training. It is shown that most "classical" second-order methods are impractical for large neural networks. A few methods are proposed that do not have these limitations.

UR - http://www.scopus.com/inward/record.url?scp=84872543023&partnerID=8YFLogxK

U2 - 10.1007/978-3-642-35289-8_3

DO - 10.1007/978-3-642-35289-8_3

M3 - Chapter

AN - SCOPUS:84872543023

SN - 9783642352881

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 9

EP - 48

BT - Neural Networks

PB - Springer Verlag

ER -

Efficient backprop

Abstract

Publication series

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this