Asymptotic statistical theory of overtraining and cross-validation

Shun Ichi Amari; Noboru Murata; Klaus Robert Müller; Michael Finke; Howard Hua Yang

doi:10.1109/72.623200

Asymptotic statistical theory of overtraining and cross-validation

Shun Ichi Amari, Noboru Murata, Klaus Robert Müller, Michael Finke, Howard Hua Yang

Research output: Contribution to journal › Article › peer-review

286 Citations (Scopus)

Abstract

A statistical theory for overtraining is proposed. The analysis treats general realizable stochastic neural networks, trained with Kullback-Leibler divergence in the asymptotic case of a large number of training examples. It is shown that the asymptotic gain in the generalization error is small if we perform early stopping, even if we have access to the optimal stopping time. Considering cross-validation stopping we answer the question: In what ratio the examples should be divided into training and cross-validation sets in order to obtain the optimum performance. Although cross-validated early stopping is useless in the asymptotic region, it surely decreases the generalization error in the nonasymptotic region. Our large scale simulations done on a CM5 are in nice agreement with our analytical findings.

Original language	English
Pages (from-to)	985-996
Number of pages	12
Journal	IEEE Transactions on Neural Networks
Volume	8
Issue number	5
DOIs	https://doi.org/10.1109/72.623200
Publication status	Published - 1997

Bibliographical note

Funding Information:
Manuscript received September 11, 1995; revised October 21, 1996 and May 10, 1997. K.-R. Müller was supported in part by the EC S & T fellowship (FTJ 3-004). This work was supported by the National Institutes of Health (P41RRO 5969) and CNCPST Paris (96JR063).

Keywords

Asymptotic analysis
Cross-validation
Early stopping
Generalization
Overtraining
Stochastic neural networks

ASJC Scopus subject areas

Software
Computer Science Applications
Computer Networks and Communications
Artificial Intelligence

Access to Document

10.1109/72.623200

Cite this

@article{b176bc2c19f64f2e8927c4f314e7067a,

title = "Asymptotic statistical theory of overtraining and cross-validation",

abstract = "A statistical theory for overtraining is proposed. The analysis treats general realizable stochastic neural networks, trained with Kullback-Leibler divergence in the asymptotic case of a large number of training examples. It is shown that the asymptotic gain in the generalization error is small if we perform early stopping, even if we have access to the optimal stopping time. Considering cross-validation stopping we answer the question: In what ratio the examples should be divided into training and cross-validation sets in order to obtain the optimum performance. Although cross-validated early stopping is useless in the asymptotic region, it surely decreases the generalization error in the nonasymptotic region. Our large scale simulations done on a CM5 are in nice agreement with our analytical findings.",

keywords = "Asymptotic analysis, Cross-validation, Early stopping, Generalization, Overtraining, Stochastic neural networks",

author = "Amari, {Shun Ichi} and Noboru Murata and M{\"u}ller, {Klaus Robert} and Michael Finke and Yang, {Howard Hua}",

note = "Funding Information: Manuscript received September 11, 1995; revised October 21, 1996 and May 10, 1997. K.-R. M{\"u}ller was supported in part by the EC S & T fellowship (FTJ 3-004). This work was supported by the National Institutes of Health (P41RRO 5969) and CNCPST Paris (96JR063).",

year = "1997",

doi = "10.1109/72.623200",

language = "English",

volume = "8",

pages = "985--996",

journal = "IEEE Transactions on Neural Networks",

issn = "1045-9227",

publisher = "IEEE Computational Intelligence Society",

number = "5",

}

TY - JOUR

T1 - Asymptotic statistical theory of overtraining and cross-validation

AU - Amari, Shun Ichi

AU - Murata, Noboru

AU - Müller, Klaus Robert

AU - Finke, Michael

AU - Yang, Howard Hua

N1 - Funding Information: Manuscript received September 11, 1995; revised October 21, 1996 and May 10, 1997. K.-R. Müller was supported in part by the EC S & T fellowship (FTJ 3-004). This work was supported by the National Institutes of Health (P41RRO 5969) and CNCPST Paris (96JR063).

PY - 1997

Y1 - 1997

N2 - A statistical theory for overtraining is proposed. The analysis treats general realizable stochastic neural networks, trained with Kullback-Leibler divergence in the asymptotic case of a large number of training examples. It is shown that the asymptotic gain in the generalization error is small if we perform early stopping, even if we have access to the optimal stopping time. Considering cross-validation stopping we answer the question: In what ratio the examples should be divided into training and cross-validation sets in order to obtain the optimum performance. Although cross-validated early stopping is useless in the asymptotic region, it surely decreases the generalization error in the nonasymptotic region. Our large scale simulations done on a CM5 are in nice agreement with our analytical findings.

AB - A statistical theory for overtraining is proposed. The analysis treats general realizable stochastic neural networks, trained with Kullback-Leibler divergence in the asymptotic case of a large number of training examples. It is shown that the asymptotic gain in the generalization error is small if we perform early stopping, even if we have access to the optimal stopping time. Considering cross-validation stopping we answer the question: In what ratio the examples should be divided into training and cross-validation sets in order to obtain the optimum performance. Although cross-validated early stopping is useless in the asymptotic region, it surely decreases the generalization error in the nonasymptotic region. Our large scale simulations done on a CM5 are in nice agreement with our analytical findings.

KW - Asymptotic analysis

KW - Cross-validation

KW - Early stopping

KW - Generalization

KW - Overtraining

KW - Stochastic neural networks

UR - http://www.scopus.com/inward/record.url?scp=0031236925&partnerID=8YFLogxK

U2 - 10.1109/72.623200

DO - 10.1109/72.623200

M3 - Article

C2 - 18255701

AN - SCOPUS:0031236925

SN - 1045-9227

VL - 8

SP - 985

EP - 996

JO - IEEE Transactions on Neural Networks

JF - IEEE Transactions on Neural Networks

IS - 5

ER -

Asymptotic statistical theory of overtraining and cross-validation

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this