Input space versus feature space in kernel-based methods

Bernhard Schölkopf; Sebastian Mika; Chris J.C. Burges; Philipp Knirsch; Klaus Robert Müller; Gunnar Rätsch; Alexander J. Smola

doi:10.1109/72.788641

Input space versus feature space in kernel-based methods

Bernhard Schölkopf, Sebastian Mika, Chris J.C. Burges, Philipp Knirsch, Klaus Robert Müller, Gunnar Rätsch, Alexander J. Smola

Research output: Contribution to journal › Article › peer-review

1042 Citations (Scopus)

Abstract

This paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector (SV) kernel functions. We first discuss the geometry of feature space. In particular, we review what is known about the shape of the image of input space under the feature space map, and how this influences the capacity of SV methods. Following this, we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel, using the example of the class of inhomogeneous polynomial kernels, which are often used in SV pattern recognition. We then discuss the connection between feature space and input space by dealing with the question of how one can, given some vector in feature space, find a preimage (exact or approximate) in input space. We describe algorithms to tackle this issue, and show their utility in two applications of kernel methods. First, we use it to reduce the computational complexity of SV decision functions; second, we combine it with the Kernel PCA algorithm, thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real-world data.

Original language	English
Pages (from-to)	1000-1017
Number of pages	18
Journal	IEEE Transactions on Neural Networks
Volume	10
Issue number	5
DOIs	https://doi.org/10.1109/72.788641
Publication status	Published - 1999

Bibliographical note

Funding Information:
Manuscript received January 23, 1999; revised May 16, 1999. Part of this work was done while P. Knirsch was with Bell Labs and B. Schölkopf and A. J. Smola were with the Department of Engineering, Australian National University, Canberra. This work was supported by the ARC and the DFG under Grant Ja 379/52,71,91. B. Schölkopf was with GMD FIRST, 12489 Berlin, Germany. He is now with Mocrosoft Research Ltd., Cambridge CB2, U.K. S. Mika, K.-R. Müller, G. Rätsch, and A. J. Smola are with GMD FIRST, 12489 Berlin, Germany. C. J. C. Burges is with Bell Laboratories, Holmdel NJ, USA. P. Knirsch is with the Max-Planck-Institut für biologische Kybernetik, 72076 Tübingen, Germany. Publisher Item Identifier S 1045-9227(99)07268-9.

ASJC Scopus subject areas

Software
Computer Science Applications
Computer Networks and Communications
Artificial Intelligence

Access to Document

10.1109/72.788641

Cite this

@article{d342e311ae47438d80a946909c272d50,

title = "Input space versus feature space in kernel-based methods",

abstract = "This paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector (SV) kernel functions. We first discuss the geometry of feature space. In particular, we review what is known about the shape of the image of input space under the feature space map, and how this influences the capacity of SV methods. Following this, we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel, using the example of the class of inhomogeneous polynomial kernels, which are often used in SV pattern recognition. We then discuss the connection between feature space and input space by dealing with the question of how one can, given some vector in feature space, find a preimage (exact or approximate) in input space. We describe algorithms to tackle this issue, and show their utility in two applications of kernel methods. First, we use it to reduce the computational complexity of SV decision functions; second, we combine it with the Kernel PCA algorithm, thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real-world data.",

author = "Bernhard Sch{\"o}lkopf and Sebastian Mika and Burges, {Chris J.C.} and Philipp Knirsch and M{\"u}ller, {Klaus Robert} and Gunnar R{\"a}tsch and Smola, {Alexander J.}",

note = "Funding Information: Manuscript received January 23, 1999; revised May 16, 1999. Part of this work was done while P. Knirsch was with Bell Labs and B. Sch{\"o}lkopf and A. J. Smola were with the Department of Engineering, Australian National University, Canberra. This work was supported by the ARC and the DFG under Grant Ja 379/52,71,91. B. Sch{\"o}lkopf was with GMD FIRST, 12489 Berlin, Germany. He is now with Mocrosoft Research Ltd., Cambridge CB2, U.K. S. Mika, K.-R. M{\"u}ller, G. R{\"a}tsch, and A. J. Smola are with GMD FIRST, 12489 Berlin, Germany. C. J. C. Burges is with Bell Laboratories, Holmdel NJ, USA. P. Knirsch is with the Max-Planck-Institut f{\"u}r biologische Kybernetik, 72076 T{\"u}bingen, Germany. Publisher Item Identifier S 1045-9227(99)07268-9.",

year = "1999",

doi = "10.1109/72.788641",

language = "English",

volume = "10",

pages = "1000--1017",

journal = "IEEE Transactions on Neural Networks",

issn = "1045-9227",

publisher = "IEEE Computational Intelligence Society",

number = "5",

}

TY - JOUR

T1 - Input space versus feature space in kernel-based methods

AU - Schölkopf, Bernhard

AU - Mika, Sebastian

AU - Burges, Chris J.C.

AU - Knirsch, Philipp

AU - Müller, Klaus Robert

AU - Rätsch, Gunnar

AU - Smola, Alexander J.

N1 - Funding Information: Manuscript received January 23, 1999; revised May 16, 1999. Part of this work was done while P. Knirsch was with Bell Labs and B. Schölkopf and A. J. Smola were with the Department of Engineering, Australian National University, Canberra. This work was supported by the ARC and the DFG under Grant Ja 379/52,71,91. B. Schölkopf was with GMD FIRST, 12489 Berlin, Germany. He is now with Mocrosoft Research Ltd., Cambridge CB2, U.K. S. Mika, K.-R. Müller, G. Rätsch, and A. J. Smola are with GMD FIRST, 12489 Berlin, Germany. C. J. C. Burges is with Bell Laboratories, Holmdel NJ, USA. P. Knirsch is with the Max-Planck-Institut für biologische Kybernetik, 72076 Tübingen, Germany. Publisher Item Identifier S 1045-9227(99)07268-9.

PY - 1999

Y1 - 1999

N2 - This paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector (SV) kernel functions. We first discuss the geometry of feature space. In particular, we review what is known about the shape of the image of input space under the feature space map, and how this influences the capacity of SV methods. Following this, we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel, using the example of the class of inhomogeneous polynomial kernels, which are often used in SV pattern recognition. We then discuss the connection between feature space and input space by dealing with the question of how one can, given some vector in feature space, find a preimage (exact or approximate) in input space. We describe algorithms to tackle this issue, and show their utility in two applications of kernel methods. First, we use it to reduce the computational complexity of SV decision functions; second, we combine it with the Kernel PCA algorithm, thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real-world data.

AB - This paper collects some ideas targeted at advancing our understanding of the feature spaces associated with support vector (SV) kernel functions. We first discuss the geometry of feature space. In particular, we review what is known about the shape of the image of input space under the feature space map, and how this influences the capacity of SV methods. Following this, we describe how the metric governing the intrinsic geometry of the mapped surface can be computed in terms of the kernel, using the example of the class of inhomogeneous polynomial kernels, which are often used in SV pattern recognition. We then discuss the connection between feature space and input space by dealing with the question of how one can, given some vector in feature space, find a preimage (exact or approximate) in input space. We describe algorithms to tackle this issue, and show their utility in two applications of kernel methods. First, we use it to reduce the computational complexity of SV decision functions; second, we combine it with the Kernel PCA algorithm, thereby constructing a nonlinear statistical denoising technique which is shown to perform well on real-world data.

UR - http://www.scopus.com/inward/record.url?scp=0032594954&partnerID=8YFLogxK

U2 - 10.1109/72.788641

DO - 10.1109/72.788641

M3 - Article

C2 - 18252603

AN - SCOPUS:0032594954

SN - 1045-9227

VL - 10

SP - 1000

EP - 1017

JO - IEEE Transactions on Neural Networks

JF - IEEE Transactions on Neural Networks

IS - 5

ER -

Input space versus feature space in kernel-based methods

Abstract

Bibliographical note

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this