An empirical study on the properties of random bases for kernel methods

Maximilian Alber; Pieter Jan Kindermans; Kristof T. Schütt; Klaus Robert Müller; Fei Sha

An empirical study on the properties of random bases for kernel methods

Maximilian Alber, Pieter Jan Kindermans, Kristof T. Schütt, Klaus Robert Müller, Fei Sha

Research output: Contribution to journal › Conference article › peer-review

Abstract

Kernel machines as well as neural networks possess universal function approximation properties. Nevertheless in practice their ways of choosing the appropriate function class differ. Specifically neural networks learn a representation by adapting their basis functions to the data and the task at hand, while kernel methods typically use a basis that is not adapted during training. In this work, we contrast random features of approximated kernel machines with learned features of neural networks. Our analysis reveals how these random and adaptive basis functions affect the quality of learning. Furthermore, we present basis adaptation schemes that allow for a more compact representation, while retaining the generalization properties of kernel machines.

Original language	English
Pages (from-to)	2764-2775
Number of pages	12
Journal	Advances in Neural Information Processing Systems
Volume	2017-December
Publication status	Published - 2017
Event	31st Annual Conference on Neural Information Processing Systems, NIPS 2017 - Long Beach, United States Duration: 2017 Dec 4 → 2017 Dec 9

Bibliographical note

Funding Information:
MA, KS, KRM, and FS acknowledge support by the Federal Ministry of Education and Research (BMBF) under 01IS14013A. PJK has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement NO 657679. KRM further acknowledges partial funding by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (No. 2017-0-00451), BK21 and by DFG. FS is partially supported by NSF IIS-1065243, 1451412, 1513966/1632803, 1208500, CCF-1139148, a Google Research Award, an Alfred. P. Sloan Research Fellowship and ARO# W911NF-12-1-0241 and W911NF-15-1-0484. This work was supported by NVIDIA with a hardware donation.

Publisher Copyright:
© 2017 Neural information processing systems foundation. All rights reserved.

ASJC Scopus subject areas

Computer Networks and Communications
Information Systems
Signal Processing

Cite this

@article{d9950dbc43aa4aee8c453cc83a347991,

title = "An empirical study on the properties of random bases for kernel methods",

abstract = "Kernel machines as well as neural networks possess universal function approximation properties. Nevertheless in practice their ways of choosing the appropriate function class differ. Specifically neural networks learn a representation by adapting their basis functions to the data and the task at hand, while kernel methods typically use a basis that is not adapted during training. In this work, we contrast random features of approximated kernel machines with learned features of neural networks. Our analysis reveals how these random and adaptive basis functions affect the quality of learning. Furthermore, we present basis adaptation schemes that allow for a more compact representation, while retaining the generalization properties of kernel machines.",

author = "Maximilian Alber and Kindermans, {Pieter Jan} and Sch{\"u}tt, {Kristof T.} and M{\"u}ller, {Klaus Robert} and Fei Sha",

note = "Funding Information: MA, KS, KRM, and FS acknowledge support by the Federal Ministry of Education and Research (BMBF) under 01IS14013A. PJK has received funding from the European Union{\textquoteright}s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement NO 657679. KRM further acknowledges partial funding by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (No. 2017-0-00451), BK21 and by DFG. FS is partially supported by NSF IIS-1065243, 1451412, 1513966/1632803, 1208500, CCF-1139148, a Google Research Award, an Alfred. P. Sloan Research Fellowship and ARO# W911NF-12-1-0241 and W911NF-15-1-0484. This work was supported by NVIDIA with a hardware donation. Publisher Copyright: {\textcopyright} 2017 Neural information processing systems foundation. All rights reserved.; 31st Annual Conference on Neural Information Processing Systems, NIPS 2017 ; Conference date: 04-12-2017 Through 09-12-2017",

year = "2017",

language = "English",

volume = "2017-December",

pages = "2764--2775",

journal = "Advances in Neural Information Processing Systems",

issn = "1049-5258",

publisher = "Neural information processing systems foundation",

}

TY - JOUR

T1 - An empirical study on the properties of random bases for kernel methods

AU - Alber, Maximilian

AU - Kindermans, Pieter Jan

AU - Schütt, Kristof T.

AU - Müller, Klaus Robert

AU - Sha, Fei

N1 - Funding Information: MA, KS, KRM, and FS acknowledge support by the Federal Ministry of Education and Research (BMBF) under 01IS14013A. PJK has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Sklodowska-Curie grant agreement NO 657679. KRM further acknowledges partial funding by the Institute for Information & Communications Technology Promotion (IITP) grant funded by the Korea government (No. 2017-0-00451), BK21 and by DFG. FS is partially supported by NSF IIS-1065243, 1451412, 1513966/1632803, 1208500, CCF-1139148, a Google Research Award, an Alfred. P. Sloan Research Fellowship and ARO# W911NF-12-1-0241 and W911NF-15-1-0484. This work was supported by NVIDIA with a hardware donation. Publisher Copyright: © 2017 Neural information processing systems foundation. All rights reserved.

PY - 2017

Y1 - 2017

N2 - Kernel machines as well as neural networks possess universal function approximation properties. Nevertheless in practice their ways of choosing the appropriate function class differ. Specifically neural networks learn a representation by adapting their basis functions to the data and the task at hand, while kernel methods typically use a basis that is not adapted during training. In this work, we contrast random features of approximated kernel machines with learned features of neural networks. Our analysis reveals how these random and adaptive basis functions affect the quality of learning. Furthermore, we present basis adaptation schemes that allow for a more compact representation, while retaining the generalization properties of kernel machines.

AB - Kernel machines as well as neural networks possess universal function approximation properties. Nevertheless in practice their ways of choosing the appropriate function class differ. Specifically neural networks learn a representation by adapting their basis functions to the data and the task at hand, while kernel methods typically use a basis that is not adapted during training. In this work, we contrast random features of approximated kernel machines with learned features of neural networks. Our analysis reveals how these random and adaptive basis functions affect the quality of learning. Furthermore, we present basis adaptation schemes that allow for a more compact representation, while retaining the generalization properties of kernel machines.

UR - http://www.scopus.com/inward/record.url?scp=85047001988&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85047001988

SN - 1049-5258

VL - 2017-December

SP - 2764

EP - 2775

JO - Advances in Neural Information Processing Systems

JF - Advances in Neural Information Processing Systems

T2 - 31st Annual Conference on Neural Information Processing Systems, NIPS 2017

Y2 - 4 December 2017 through 9 December 2017

ER -

An empirical study on the properties of random bases for kernel methods

Abstract

Bibliographical note

ASJC Scopus subject areas

Other files and links

Fingerprint

Cite this