Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors

Koji Tsuda; Masashi Sugiyama; Klaus Robert Müller

doi:10.1109/72.977272

Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors

Koji Tsuda, Masashi Sugiyama, Klaus Robert Müller

Research output: Contribution to journal › Article › peer-review

8 Citations (Scopus)

Abstract

Nonquadratic regularizers, in particular the l₁ norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l₁ norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.

Original language	English
Pages (from-to)	70-80
Number of pages	11
Journal	IEEE Transactions on Neural Networks
Volume	13
Issue number	1
DOIs	https://doi.org/10.1109/72.977272
Publication status	Published - 2002 Jan

Bibliographical note

Funding Information:
Manuscript received November 8, 2000; revised June 27, 2001. The work of K.-R. Müller was supported in part by DFG under Contracts JA 379/91, MU 987/11 and the EU in the Neurocolt 2, and the BLISS Project (IST-1999-14190). K. Tsuda is with Fraunhofer FIRST, 12489 Berlin, Germany. He is also with AIST Computational Biology Research Center, Tokyo 135-0064, Japan (e-mail: [email protected]). M. Sugiyama is with the Tokyo Institute of Technology, Tokyo 152-8552, Japan (e-mail: [email protected]). K.-R. Müller is with Fraunhofer FIRST, 12489 Berlin, Germany. He is also with the University of Potsdam, 14469 Potsdam, Germany (e-mail: [email protected]). Publisher Item Identifier S 1045-9227(02)00353-3.

Keywords

Kernel methods
Model selection
Regularization
Sparse regressors
Subspace information criterion (SIC)

ASJC Scopus subject areas

Software
Computer Science Applications
Computer Networks and Communications
Artificial Intelligence

Access to Document

10.1109/72.977272

Cite this

@article{20993473d485492cb694ef052013cc9a,

title = "Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors",

abstract = "Nonquadratic regularizers, in particular the l1 norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l1 norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.",

keywords = "Kernel methods, Model selection, Regularization, Sparse regressors, Subspace information criterion (SIC)",

author = "Koji Tsuda and Masashi Sugiyama and M{\"u}ller, {Klaus Robert}",

note = "Funding Information: Manuscript received November 8, 2000; revised June 27, 2001. The work of K.-R. M{\"u}ller was supported in part by DFG under Contracts JA 379/91, MU 987/11 and the EU in the Neurocolt 2, and the BLISS Project (IST-1999-14190). K. Tsuda is with Fraunhofer FIRST, 12489 Berlin, Germany. He is also with AIST Computational Biology Research Center, Tokyo 135-0064, Japan (e-mail: [email protected]). M. Sugiyama is with the Tokyo Institute of Technology, Tokyo 152-8552, Japan (e-mail: [email protected]). K.-R. M{\"u}ller is with Fraunhofer FIRST, 12489 Berlin, Germany. He is also with the University of Potsdam, 14469 Potsdam, Germany (e-mail: [email protected]). Publisher Item Identifier S 1045-9227(02)00353-3.",

year = "2002",

month = jan,

doi = "10.1109/72.977272",

language = "English",

volume = "13",

pages = "70--80",

journal = "IEEE Transactions on Neural Networks",

issn = "1045-9227",

publisher = "IEEE Computational Intelligence Society",

number = "1",

}

TY - JOUR

T1 - Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors

AU - Tsuda, Koji

AU - Sugiyama, Masashi

AU - Müller, Klaus Robert

N1 - Funding Information: Manuscript received November 8, 2000; revised June 27, 2001. The work of K.-R. Müller was supported in part by DFG under Contracts JA 379/91, MU 987/11 and the EU in the Neurocolt 2, and the BLISS Project (IST-1999-14190). K. Tsuda is with Fraunhofer FIRST, 12489 Berlin, Germany. He is also with AIST Computational Biology Research Center, Tokyo 135-0064, Japan (e-mail: [email protected]). M. Sugiyama is with the Tokyo Institute of Technology, Tokyo 152-8552, Japan (e-mail: [email protected]). K.-R. Müller is with Fraunhofer FIRST, 12489 Berlin, Germany. He is also with the University of Potsdam, 14469 Potsdam, Germany (e-mail: [email protected]). Publisher Item Identifier S 1045-9227(02)00353-3.

PY - 2002/1

Y1 - 2002/1

N2 - Nonquadratic regularizers, in particular the l1 norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l1 norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.

AB - Nonquadratic regularizers, in particular the l1 norm regularizer can yield sparse solutions that generalize well. In this work we propose the generalized subspace information criterion (GSIC) that allows to predict the generalization error for this useful family of regularizers. We show that under some technical assumptions GSIC is an asymptotically unbiased estimator of the generalization error. GSIC is demonstrated to have a good performance in experiments with the l1 norm regularizer as we compare with the network information criterion (NIC) and cross- validation in relatively large sample cases. However in the small sample case, GSIC tends to fail to capture the optimal model due to its large variance. Therefore, also a biased version of GSIC is introduced, which achieves reliable model selection in the relevant and challenging scenario of high-dimensional data and few samples.

KW - Kernel methods

KW - Model selection

KW - Regularization

KW - Sparse regressors

KW - Subspace information criterion (SIC)

UR - http://www.scopus.com/inward/record.url?scp=0036130853&partnerID=8YFLogxK

U2 - 10.1109/72.977272

DO - 10.1109/72.977272

M3 - Article

C2 - 18244410

AN - SCOPUS:0036130853

SN - 1045-9227

VL - 13

SP - 70

EP - 80

JO - IEEE Transactions on Neural Networks

JF - IEEE Transactions on Neural Networks

IS - 1

ER -

Subspace information criterion for nonquadratic regularizers-model selection for sparse regressors

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this