Package 'kernlab'

Description

as.kernelMatrix in package kernlab can be used to coerce the kernelMatrix class to matrix objects representing a kernel matrix. These matrices can then be used with the kernelMatrix interfaces which most of the functions in kernlab support.

Usage

## S4 method for signature 'matrix'
as.kernelMatrix(x, center = FALSE)

Arguments

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

kernelMatrix, dots

Examples

## Create toy data
x <- rbind(matrix(rnorm(10),,2),matrix(rnorm(10,mean=3),,2))
y <- matrix(c(rep(1,5),rep(-1,5)))

### Use as.kernelMatrix to label the cov. matrix as a kernel matrix
### which is eq. to using a linear kernel 

K <- as.kernelMatrix(crossprod(t(x)))

K

svp2 <- ksvm(K, y, type="C-svc")

svp2

Description

couple is used to link class-probability estimates produced by pairwise coupling in multi-class classification problems.

Usage

couple(probin, coupler = "minpair")

Arguments

Details

As binary classification problems are much easier to solve many techniques exist to decompose multi-class classification problems into many binary classification problems (voting, error codes, etc.). Pairwise coupling (one against one) constructs a rule for discriminating between every pair of classes and then selecting the class with the most winning two-class decisions. By using Platt's probabilities output for SVM one can get a class probability for each of the $k(k-1)/2$ models created in the pairwise classification. The couple method implements various techniques to combine these probabilities.

Value

A matrix with the resulting probability estimates.

Author(s)

Alexandros Karatzoglou
[email protected]

References

Ting-Fan Wu, Chih-Jen Lin, ruby C. Weng
Probability Estimates for Multi-class Classification by Pairwise Coupling
Neural Information Processing Symposium 2003
https://papers.neurips.cc/paper/2454-probability-estimates-for-multi-class-classification-by-pairwise-coupling.pdf

See Also

predict.ksvm, ksvm

Examples

## create artificial pairwise probabilities
pairs <- matrix(c(0.82,0.12,0.76,0.1,0.9,0.05),2)

couple(pairs)

couple(pairs, coupler="pkpd")

couple(pairs, coupler ="vote")

Description

The csi function in kernlab is an implementation of an incomplete Cholesky decomposition algorithm which exploits side information (e.g., classification labels, regression responses) to compute a low rank decomposition of a kernel matrix from the data.

Usage

## S4 method for signature 'matrix'
csi(x, y, kernel="rbfdot", kpar=list(sigma=0.1), rank,
centering = TRUE, kappa = 0.99 ,delta = 40 ,tol = 1e-5)

Arguments

Details

An incomplete cholesky decomposition calculates $Z$ where $K= ZZ'$ $K$ being the kernel matrix. Since the rank of a kernel matrix is usually low, $Z$ tends to be smaller then the complete kernel matrix. The decomposed matrix can be used to create memory efficient kernel-based algorithms without the need to compute and store a complete kernel matrix in memory.
csi uses the class labels, or regression responses to compute a more appropriate approximation for the problem at hand considering the additional information from the response variable.

Value

An S4 object of class "csi" which is an extension of the class "matrix". The object is the decomposed kernel matrix along with the slots :

slots can be accessed either by object@slot or by accessor functions with the same name (e.g., pivots(object))

Author(s)

Alexandros Karatzoglou (based on Matlab code by Francis Bach)
[email protected]

References

Francis R. Bach, Michael I. Jordan
Predictive low-rank decomposition for kernel methods.
Proceedings of the Twenty-second International Conference on Machine Learning (ICML) 2005
http://www.di.ens.fr/~fbach/bach_jordan_csi.pdf

See Also

inchol, chol, csi-class

Examples

data(iris)

## create multidimensional y matrix
yind <- t(matrix(1:3,3,150))
ymat <- matrix(0, 150, 3)
ymat[yind==as.integer(iris[,5])] <- 1

datamatrix <- as.matrix(iris[,-5])
# initialize kernel function
rbf <- rbfdot(sigma=0.1)
rbf
Z <- csi(datamatrix,ymat, kernel=rbf, rank = 30)
dim(Z)
pivots(Z)
# calculate kernel matrix
K <- crossprod(t(Z))
# difference between approximated and real kernel matrix
(K - kernelMatrix(kernel=rbf, datamatrix))[6,]

Description

The reduced Cholesky decomposition object

Objects from the Class

Objects can be created by calls of the form new("csi", ...). or by calling the csi function.

Slots

.Data:

Object of class "matrix" contains the decomposed matrix

pivots:

Object of class "vector" contains the pivots performed

diagresidues:

Object of class "vector" contains the diagonial residues

maxresiduals:

Object of class "vector" contains the maximum residues

predgain

Object of class "vector" contains the predicted gain before adding each column

truegain

Object of class "vector" contains the actual gain after adding each column

Q

Object of class "matrix" contains Q from the QR decomposition of the kernel matrix

R

Object of class "matrix" contains R from the QR decomposition of the kernel matrix

Extends

Class "matrix", directly.

Methods

diagresidues

signature(object = "csi"): returns the diagonial residues

maxresiduals

signature(object = "csi"): returns the maximum residues

pivots

signature(object = "csi"): returns the pivots performed

predgain

signature(object = "csi"): returns the predicted gain before adding each column

truegain

signature(object = "csi"): returns the actual gain after adding each column

Q

signature(object = "csi"): returns Q from the QR decomposition of the kernel matrix

R

signature(object = "csi"): returns R from the QR decomposition of the kernel matrix

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

csi, inchol-class

Examples

data(iris)

## create multidimensional y matrix
yind <- t(matrix(1:3,3,150))
ymat <- matrix(0, 150, 3)
ymat[yind==as.integer(iris[,5])] <- 1

datamatrix <- as.matrix(iris[,-5])
# initialize kernel function
rbf <- rbfdot(sigma=0.1)
rbf
Z <- csi(datamatrix,ymat, kernel=rbf, rank = 30)
dim(Z)
pivots(Z)
# calculate kernel matrix
K <- crossprod(t(Z))
# difference between approximated and real kernel matrix
(K - kernelMatrix(kernel=rbf, datamatrix))[6,]

Description

The kernel generating functions provided in kernlab.
The Gaussian RBF kernel $k(x,x') = \exp(-\sigma \|x - x'\|^2)$
The Polynomial kernel $k(x,x') = (scale <x, x'> + offset)^{degree}$
The Linear kernel $k(x,x') = <x, x'>$
The Hyperbolic tangent kernel $k(x, x') = \tanh(scale <x, x'> + offset)$
The Laplacian kernel $k(x,x') = \exp(-\sigma \|x - x'\|)$
The Bessel kernel $k(x,x') = (- Bessel_{(\nu+1)}^n \sigma \|x - x'\|^2)$
The ANOVA RBF kernel $k(x,x') = \sum_{1\leq i_1 \ldots < i_D \leq N} \prod_{d=1}^D k(x_{id}, {x'}_{id})$ where k(x,x) is a Gaussian RBF kernel.
The Spline kernel $\prod_{d=1}^D 1 + x_i x_j + x_i x_j min(x_i, x_j) - \frac{x_i + x_j}{2} min(x_i,x_j)^2 + \frac{min(x_i,x_j)^3}{3}$ \ The String kernels (see stringdot.

Usage

rbfdot(sigma = 1)

polydot(degree = 1, scale = 1, offset = 1)

tanhdot(scale = 1, offset = 1)

vanilladot()

laplacedot(sigma = 1)

besseldot(sigma = 1, order = 1, degree = 1)

anovadot(sigma = 1, degree = 1)

splinedot()

Arguments

Details

The kernel generating functions are used to initialize a kernel function which calculates the dot (inner) product between two feature vectors in a Hilbert Space. These functions can be passed as a kernel argument on almost all functions in kernlab(e.g., ksvm, kpca etc).

Although using one of the existing kernel functions as a kernel argument in various functions in kernlab has the advantage that optimized code is used to calculate various kernel expressions, any other function implementing a dot product of class kernel can also be used as a kernel argument. This allows the user to use, test and develop special kernels for a given data set or algorithm. For details on the string kernels see stringdot.

Value

Return an S4 object of class kernel which extents the function class. The resulting function implements the given kernel calculating the inner (dot) product between two vectors.

The kernel parameters can be accessed by the kpar function.

Note

If the offset in the Polynomial kernel is set to $0$, we obtain homogeneous polynomial kernels, for positive values, we have inhomogeneous kernels. Note that for negative values the kernel does not satisfy Mercer's condition and thus the optimizers may fail.

In the Hyperbolic tangent kernel if the offset is negative the likelihood of obtaining a kernel matrix that is not positive definite is much higher (since then even some diagonal elements may be negative), hence if this kernel has to be used, the offset should always be positive. Note, however, that this is no guarantee that the kernel will be positive.

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

stringdot, kernelMatrix , kernelMult, kernelPol

Examples

rbfkernel <- rbfdot(sigma = 0.1)
rbfkernel

kpar(rbfkernel)

## create two vectors
x <- rnorm(10)
y <- rnorm(10)

## calculate dot product
rbfkernel(x,y)

Description

gausspr is an implementation of Gaussian processes for classification and regression.

Usage

## S4 method for signature 'formula'
gausspr(x, data=NULL, ..., subset, na.action = na.omit, scaled = TRUE)

## S4 method for signature 'vector'
gausspr(x,...)

## S4 method for signature 'matrix'
gausspr(x, y, scaled = TRUE, type= NULL, kernel="rbfdot",
          kpar="automatic", var=1, variance.model = FALSE, tol=0.0005,
          cross=0, fit=TRUE, ... , subset, na.action = na.omit)

Arguments

Details

A Gaussian process is specified by a mean and a covariance function. The mean is a function of $x$ (which is often the zero function), and the covariance is a function $C(x,x')$ which expresses the expected covariance between the value of the function $y$ at the points $x$ and $x'$. The actual function $y(x)$ in any data modeling problem is assumed to be a single sample from this Gaussian distribution. Laplace approximation is used for the parameter estimation in gaussian processes for classification.

The predict function can return class probabilities for classification problems by setting the type parameter to "probabilities". For the regression setting the type parameter to "variance" or "sdeviation" returns the estimated variance or standard deviation at each predicted point.

Value

An S4 object of class "gausspr" containing the fitted model along with information. Accessor functions can be used to access the slots of the object which include :

Author(s)

Alexandros Karatzoglou
[email protected]

References

C. K. I. Williams and D. Barber
Bayesian classification with Gaussian processes.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12):1342-1351, 1998
https://homepages.inf.ed.ac.uk/ckiw/postscript/pami_final.ps.gz

See Also

predict.gausspr, rvm, ksvm, gausspr-class, lssvm

Examples

# train model
data(iris)
test <- gausspr(Species~.,data=iris,var=2)
test
alpha(test)

# predict on the training set
predict(test,iris[,-5])
# class probabilities 
predict(test, iris[,-5], type="probabilities")

# create regression data
x <- seq(-20,20,0.1)
y <- sin(x)/x + rnorm(401,sd=0.03)

# regression with gaussian processes
foo <- gausspr(x, y)
foo

# predict and plot
ytest <- predict(foo, x)
plot(x, y, type ="l")
lines(x, ytest, col="red")


#predict and variance
x = c(-4, -3, -2, -1,  0, 0.5, 1, 2)
y = c(-2,  0,  -0.5,1,  2, 1, 0, -1)
plot(x,y)
foo2 <- gausspr(x, y, variance.model = TRUE)
xtest <- seq(-4,2,0.2)
lines(xtest, predict(foo2, xtest))
lines(xtest,
      predict(foo2, xtest)+2*predict(foo2,xtest, type="sdeviation"),
      col="red")
lines(xtest,
      predict(foo2, xtest)-2*predict(foo2,xtest, type="sdeviation"),
      col="red")

Description

The Gaussian Processes object class

Objects from the Class

Objects can be created by calls of the form new("gausspr", ...). or by calling the gausspr function

Slots

tol:

Object of class "numeric" contains tolerance of termination criteria

kernelf:

Object of class "kfunction" contains the kernel function used

kpar:

Object of class "list" contains the kernel parameter used

kcall:

Object of class "list" contains the used function call

type:

Object of class "character" contains type of problem

terms:

Object of class "ANY" contains the terms representation of the symbolic model used (when using a formula)

xmatrix:

Object of class "input" containing the data matrix used

ymatrix:

Object of class "output" containing the response matrix

fitted:

Object of class "output" containing the fitted values

lev:

Object of class "vector" containing the levels of the response (in case of classification)

nclass:

Object of class "numeric" containing the number of classes (in case of classification)

alpha:

Object of class "listI" containing the computes alpha values

alphaindex

Object of class "list" containing the indexes for the alphas in various classes (in multi-class problems).

sol

Object of class "matrix" containing the solution to the Gaussian Process formulation, it is used to compute the variance in regression problems.

scaling

Object of class "ANY" containing the scaling coefficients of the data (when case scaled = TRUE is used).

nvar:

Object of class "numeric" containing the computed variance

error:

Object of class "numeric" containing the training error

cross:

Object of class "numeric" containing the cross validation error

n.action:

Object of class "ANY" containing the action performed in NA

Methods

alpha

signature(object = "gausspr"): returns the alpha vector

cross

signature(object = "gausspr"): returns the cross validation error

error

signature(object = "gausspr"): returns the training error

fitted

signature(object = "vm"): returns the fitted values

kcall

signature(object = "gausspr"): returns the call performed

kernelf

signature(object = "gausspr"): returns the kernel function used

kpar

signature(object = "gausspr"): returns the kernel parameter used

lev

signature(object = "gausspr"): returns the response levels (in classification)

type

signature(object = "gausspr"): returns the type of problem

xmatrix

signature(object = "gausspr"): returns the data matrix used

ymatrix

signature(object = "gausspr"): returns the response matrix used

scaling

signature(object = "gausspr"): returns the scaling coefficients of the data (when scaled = TRUE is used)

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

gausspr, ksvm-class, vm-class

Examples

# train model
data(iris)
test <- gausspr(Species~.,data=iris,var=2)
test
alpha(test)
error(test)
lev(test)

Description

inchol computes the incomplete Cholesky decomposition of the kernel matrix from a data matrix.

Usage

inchol(x, kernel="rbfdot", kpar=list(sigma=0.1), tol = 0.001, 
            maxiter = dim(x)[1], blocksize = 50, verbose = 0)

Arguments

Details

An incomplete cholesky decomposition calculates $Z$ where $K= ZZ'$ $K$ being the kernel matrix. Since the rank of a kernel matrix is usually low, $Z$ tends to be smaller then the complete kernel matrix. The decomposed matrix can be used to create memory efficient kernel-based algorithms without the need to compute and store a complete kernel matrix in memory.

Value

An S4 object of class "inchol" which is an extension of the class "matrix". The object is the decomposed kernel matrix along with the slots :

slots can be accessed either by object@slot or by accessor functions with the same name (e.g., pivots(object))

Author(s)

Alexandros Karatzoglou (based on Matlab code by S.V.N. (Vishy) Vishwanathan and Alex Smola)
[email protected]

References

Francis R. Bach, Michael I. Jordan
Kernel Independent Component Analysis
Journal of Machine Learning Research 3, 1-48
https://www.jmlr.org/papers/volume3/bach02a/bach02a.pdf

See Also

csi, inchol-class, chol

Examples

data(iris)
datamatrix <- as.matrix(iris[,-5])
# initialize kernel function
rbf <- rbfdot(sigma=0.1)
rbf
Z <- inchol(datamatrix,kernel=rbf)
dim(Z)
pivots(Z)
# calculate kernel matrix
K <- crossprod(t(Z))
# difference between approximated and real kernel matrix
(K - kernelMatrix(kernel=rbf, datamatrix))[6,]

Description

The reduced Cholesky decomposition object

Objects from the Class

Objects can be created by calls of the form new("inchol", ...). or by calling the inchol function.

Slots

.Data:

Object of class "matrix" contains the decomposed matrix

pivots:

Object of class "vector" contains the pivots performed

diagresidues:

Object of class "vector" contains the diagonial residues

maxresiduals:

Object of class "vector" contains the maximum residues

Extends

Class "matrix", directly.

Methods

diagresidues

signature(object = "inchol"): returns the diagonial residues

maxresiduals

signature(object = "inchol"): returns the maximum residues

pivots

signature(object = "inchol"): returns the pivots performed

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

inchol, csi-class, csi

Examples

data(iris)
datamatrix <- as.matrix(iris[,-5])
# initialize kernel function
rbf <- rbfdot(sigma=0.1)
rbf
Z <- inchol(datamatrix,kernel=rbf)
dim(Z)
pivots(Z)
diagresidues(Z)
maxresiduals(Z)

Description

Customer Income Data from a marketing survey.

Usage

data(income)

Format

A data frame with 14 categorical variables (8993 observations).

Explanation of the variable names:

Details

A total of N=9409 questionnaires containing 502 questions were filled out by shopping mall customers in the San Francisco Bay area. The dataset is an extract from this survey. It consists of 14 demographic attributes. The dataset is a mixture of nominal and ordinal variables with a lot of missing data. The goal is to predict the Anual Income of Household from the other 13 demographics attributes.

Source

Impact Resources, Inc., Columbus, OH (1987).

Description

Online Kernel Algorithm object onlearn initialization function.

Usage

## S4 method for signature 'numeric'
inlearn(d, kernel = "rbfdot", kpar = list(sigma = 0.1),
        type = "novelty", buffersize = 1000)

Arguments

Details

The inlearn is used to initialize a blank onlearn object.

Value

The function returns an S4 object of class onlearn that can be used by the onlearn function.

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

onlearn, onlearn-class

Examples

## create toy data set
x <- rbind(matrix(rnorm(100),,2),matrix(rnorm(100)+3,,2))
y <- matrix(c(rep(1,50),rep(-1,50)),,1)

## initialize onlearn object
on <- inlearn(2, kernel = "rbfdot", kpar = list(sigma = 0.2),
              type = "classification")

## learn one data point at the time
for(i in sample(1:100,100))
on <- onlearn(on,x[i,],y[i],nu=0.03,lambda=0.1)

sign(predict(on,x))

Description

ipop solves the quadratic programming problem :
$\min(c'*x + 1/2 * x' * H * x)$
subject to:
$b <= A * x <= b + r$
$l <= x <= u$

Usage

ipop(c, H, A, b, l, u, r, sigf = 7, maxiter = 40, margin = 0.05,
     bound = 10, verb = 0)

Arguments

Details

ipop uses an interior point method to solve the quadratic programming problem.
The $H$ matrix can also be provided in the decomposed form $Z$ where $ZZ' = H$ in that case the Sherman Morrison Woodbury formula is used internally.

Value

An S4 object with the following slots

all slots can be accessed through accessor functions (see example)

Author(s)

Alexandros Karatzoglou (based on Matlab code by Alex Smola)
[email protected]

References

R. J. Vanderbei
LOQO: An interior point code for quadratic programming
Optimization Methods and Software 11, 451-484, 1999
https://vanderbei.princeton.edu/ps/loqo5.pdf

See Also

solve.QP, inchol, csi

Examples

## solve the Support Vector Machine optimization problem
data(spam)

## sample a scaled part (500 points) of the spam data set
m <- 500
set <- sample(1:dim(spam)[1],m)
x <- scale(as.matrix(spam[,-58]))[set,]
y <- as.integer(spam[set,58])
y[y==2] <- -1

##set C parameter and kernel
C <- 5
rbf <- rbfdot(sigma = 0.1)

## create H matrix etc.
H <- kernelPol(rbf,x,,y)
c <- matrix(rep(-1,m))
A <- t(y)
b <- 0
l <- matrix(rep(0,m))
u <- matrix(rep(C,m))
r <- 0

sv <- ipop(c,H,A,b,l,u,r)
sv
dual(sv)

Description

The quadratic problem solver class

Objects from the Class

Objects can be created by calls of the form new("ipop", ...). or by calling the ipop function.

Slots

primal:

Object of class "vector" the primal solution of the problem

dual:

Object of class "numeric" the dual of the problem

how:

Object of class "character" convergence information

Methods

primal

signature(object = "ipop"): Return the primal of the problem

dual

signature(object = "ipop"): Return the dual of the problem

how

signature(object = "ipop"): Return information on convergence

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

ipop

Examples

## solve the Support Vector Machine optimization problem
data(spam)

## sample a scaled part (300 points) of the spam data set
m <- 300
set <- sample(1:dim(spam)[1],m)
x <- scale(as.matrix(spam[,-58]))[set,]
y <- as.integer(spam[set,58])
y[y==2] <- -1

##set C parameter and kernel
C <- 5
rbf <- rbfdot(sigma = 0.1)

## create H matrix etc.
H <- kernelPol(rbf,x,,y)
c <- matrix(rep(-1,m))
A <- t(y)
b <- 0
l <- matrix(rep(0,m))
u <- matrix(rep(C,m))
r <- 0

sv <- ipop(c,H,A,b,l,u,r)
primal(sv)
dual(sv)
how(sv)

Description

Computes the canonical correlation analysis in feature space.

Usage

## S4 method for signature 'matrix'
kcca(x, y, kernel="rbfdot", kpar=list(sigma=0.1),
gamma = 0.1, ncomps = 10, ...)

Arguments

Details

The kernel version of canonical correlation analysis. Kernel Canonical Correlation Analysis (KCCA) is a non-linear extension of CCA. Given two random variables, KCCA aims at extracting the information which is shared by the two random variables. More precisely given $x$ and $y$ the purpose of KCCA is to provide nonlinear mappings $f(x)$ and $g(y)$ such that their correlation is maximized.

Value

An S4 object containing the following slots:

Author(s)

Alexandros Karatzoglou
[email protected]

References

Malte Kuss, Thore Graepel
The Geometry Of Kernel Canonical Correlation Analysis
https://www.microsoft.com/en-us/research/publication/the-geometry-of-kernel-canonical-correlation-analysis/

See Also

cancor, kpca, kfa, kha

Examples

## dummy data
x <- matrix(rnorm(30),15)
y <- matrix(rnorm(30),15)

kcca(x,y,ncomps=2)

Description

The "kcca" class

Objects from the Class

Objects can be created by calls of the form new("kcca", ...). or by the calling the kcca function.

Slots

kcor:

Object of class "vector" describing the correlations

xcoef:

Object of class "matrix" estimated coefficients for the x variables

ycoef:

Object of class "matrix" estimated coefficients for the y variables

Methods

kcor

signature(object = "kcca"): returns the correlations

xcoef

signature(object = "kcca"): returns the estimated coefficients for the x variables

ycoef

signature(object = "kcca"): returns the estimated coefficients for the y variables

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

kcca, kpca-class

Examples

## dummy data
x <- matrix(rnorm(30),15)
y <- matrix(rnorm(30),15)

kcca(x,y,ncomps=2)

Description

The built-in kernel classes in kernlab

Objects from the Class

Objects can be created by calls of the form new("rbfkernel"), new{"polykernel"}, new{"tanhkernel"}, new{"vanillakernel"}, new{"anovakernel"}, new{"besselkernel"}, new{"laplacekernel"}, new{"splinekernel"}, new{"stringkernel"}

or by calling the rbfdot, polydot, tanhdot, vanilladot, anovadot, besseldot, laplacedot, splinedot, stringdot functions etc..

Slots

.Data:

Object of class "function" containing the kernel function

kpar:

Object of class "list" containing the kernel parameters

Extends

Class "kernel", directly. Class "function", by class "kernel".

Methods

kernelMatrix

signature(kernel = "rbfkernel", x = "matrix"): computes the kernel matrix

kernelMult

signature(kernel = "rbfkernel", x = "matrix"): computes the quadratic kernel expression

kernelPol

signature(kernel = "rbfkernel", x = "matrix"): computes the kernel expansion

kernelFast

signature(kernel = "rbfkernel", x = "matrix"),,a: computes parts or the full kernel matrix, mainly used in kernel algorithms where columns of the kernel matrix are computed per invocation

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

dots

Examples

rbfkernel <- rbfdot(sigma = 0.1)
rbfkernel
is(rbfkernel)
kpar(rbfkernel)

Description

kernelMatrix calculates the kernel matrix $K_{ij} = k(x_i,x_j)$ or $K_{ij} = k(x_i,y_j)$.
kernelPol computes the quadratic kernel expression $H = z_i z_j k(x_i,x_j)$, $H = z_i k_j k(x_i,y_j)$.
kernelMult calculates the kernel expansion $f(x_i) = \sum_{i=1}^m z_i k(x_i,x_j)$
kernelFast computes the kernel matrix, identical to kernelMatrix, except that it also requires the squared norm of the first argument as additional input, useful in iterative kernel matrix calculations.

Usage

## S4 method for signature 'kernel'
kernelMatrix(kernel, x, y = NULL)

## S4 method for signature 'kernel'
kernelPol(kernel, x, y = NULL, z, k = NULL)

## S4 method for signature 'kernel'
kernelMult(kernel, x, y = NULL, z, blocksize = 256)

## S4 method for signature 'kernel'
kernelFast(kernel, x, y, a)

Arguments

Details

Common functions used during kernel based computations.
The kernel parameter can be set to any function, of class kernel, which computes the inner product in feature space between two vector arguments. kernlab provides the most popular kernel functions which can be initialized by using the following functions:

• rbfdot Radial Basis kernel function

• polydot Polynomial kernel function

• vanilladot Linear kernel function

• tanhdot Hyperbolic tangent kernel function

• laplacedot Laplacian kernel function

• besseldot Bessel kernel function

• anovadot ANOVA RBF kernel function

• splinedot the Spline kernel

(see example.)

kernelFast is mainly used in situations where columns of the kernel matrix are computed per invocation. In these cases, evaluating the norm of each row-entry over and over again would cause significant computational overhead.

Value

kernelMatrix returns a symmetric diagonal semi-definite matrix.
kernelPol returns a matrix.
kernelMult usually returns a one-column matrix.

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

rbfdot, polydot, tanhdot, vanilladot

Examples

## use the spam data
data(spam)
dt <- as.matrix(spam[c(10:20,3000:3010),-58])

## initialize kernel function 
rbf <- rbfdot(sigma = 0.05)
rbf

## calculate kernel matrix
kernelMatrix(rbf, dt)

yt <- as.matrix(as.integer(spam[c(10:20,3000:3010),58]))
yt[yt==2] <- -1

## calculate the quadratic kernel expression
kernelPol(rbf, dt, ,yt)

## calculate the kernel expansion
kernelMult(rbf, dt, ,yt)

Description

The Kernel Feature Analysis algorithm is an algorithm for extracting structure from possibly high-dimensional data sets. Similar to kpca a new basis for the data is found. The data can then be projected on the new basis.

Usage

## S4 method for signature 'formula'
kfa(x, data = NULL, na.action = na.omit, ...)

## S4 method for signature 'matrix'
kfa(x, kernel = "rbfdot", kpar = list(sigma = 0.1),
   features = 0, subset = 59, normalize = TRUE, na.action = na.omit)

Arguments

Details

Kernel Feature analysis is similar to Kernel PCA, but instead of extracting eigenvectors of the training dataset in feature space, it approximates the eigenvectors by selecting training patterns which are good basis vectors for the training set. It works by choosing a fixed size subset of the data set and scaling it to unit length (under the kernel). It then chooses the features that maximize the value of the inner product (kernel function) with the rest of the patterns.

Value

kfa returns an object of class kfa containing the features selected by the algorithm.

The predict function can be used to embed new data points into to the selected feature base.

Author(s)

Alexandros Karatzoglou
[email protected]

References

Alex J. Smola, Olvi L. Mangasarian and Bernhard Schoelkopf
Sparse Kernel Feature Analysis
Data Mining Institute Technical Report 99-04, October 1999
ftp://ftp.cs.wisc.edu/pub/dmi/tech-reports/99-04.ps

See Also

kpca, kfa-class

Examples

data(promotergene)
f <- kfa(~.,data=promotergene,features=2,kernel="rbfdot",
         kpar=list(sigma=0.01))
plot(predict(f,promotergene),col=as.numeric(promotergene[,1]))

Description

The class of the object returned by the Kernel Feature Analysis kfa function

Objects from the Class

Objects can be created by calls of the form new("kfa", ...) or by calling the kfa method. The objects contain the features along with the alpha values.

Slots

alpha:

Object of class "matrix" containing the alpha values

alphaindex:

Object of class "vector" containing the indexes of the selected feature

kernelf:

Object of class "kfunction" containing the kernel function used

xmatrix:

Object of class "matrix" containing the selected features

kcall:

Object of class "call" containing the kfa function call

terms:

Object of class "ANY" containing the formula terms

Methods

alpha

signature(object = "kfa"): returns the alpha values

alphaindex

signature(object = "kfa"): returns the index of the selected features

kcall

signature(object = "kfa"): returns the function call

kernelf

signature(object = "kfa"): returns the kernel function used

predict

signature(object = "kfa"): used to embed more data points to the feature base

xmatrix

signature(object = "kfa"): returns the selected features.

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

kfa, kpca-class

Examples

data(promotergene)
f <- kfa(~.,data=promotergene)

Description

Kernel Hebbian Algorithm is a nonlinear iterative algorithm for principal component analysis.

Usage

## S4 method for signature 'formula'
kha(x, data = NULL, na.action, ...)

## S4 method for signature 'matrix'
kha(x, kernel = "rbfdot", kpar = list(sigma = 0.1), features = 5, 
         eta = 0.005, th = 1e-4, maxiter = 10000, verbose = FALSE,
        na.action = na.omit, ...)

Arguments

Details

The original form of KPCA can only be used on small data sets since it requires the estimation of the eigenvectors of a full kernel matrix. The Kernel Hebbian Algorithm iteratively estimates the Kernel Principal Components with only linear order memory complexity. (see ref. for more details)

Value

An S4 object containing the principal component vectors along with the corresponding normalization values.

all the slots of the object can be accessed by accessor functions.

Note

The predict function can be used to embed new data on the new space

Author(s)

Alexandros Karatzoglou
[email protected]

References

Kwang In Kim, M.O. Franz and B. Schölkopf
Kernel Hebbian Algorithm for Iterative Kernel Principal Component Analysis
Max-Planck-Institut für biologische Kybernetik, Tübingen (109)
https://is.mpg.de/fileadmin/user_upload/files/publications/pdf2302.pdf

See Also

kpca, kfa, kcca, pca

Examples

# another example using the iris
data(iris)
test <- sample(1:150,70)

kpc <- kha(~.,data=iris[-test,-5],kernel="rbfdot",
           kpar=list(sigma=0.2),features=2, eta=0.001, maxiter=65)

#print the principal component vectors
pcv(kpc)

#plot the data projection on the components
plot(predict(kpc,iris[,-5]),col=as.integer(iris[,5]),
     xlab="1st Principal Component",ylab="2nd Principal Component")

Description

The Kernel Hebbian Algorithm class

Objects objects of class "kha"

Objects can be created by calls of the form new("kha", ...). or by calling the kha function.

Slots

pcv:

Object of class "matrix" containing the principal component vectors

eig:

Object of class "vector" containing the corresponding normalization values

eskm:

Object of class "vector" containing the kernel sum

kernelf:

Object of class "kfunction" containing the kernel function used

kpar:

Object of class "list" containing the kernel parameters used

xmatrix:

Object of class "matrix" containing the data matrix used

kcall:

Object of class "ANY" containing the function call

n.action:

Object of class "ANY" containing the action performed on NA

Methods

eig

signature(object = "kha"): returns the normalization values

kcall

signature(object = "kha"): returns the performed call

kernelf

signature(object = "kha"): returns the used kernel function

pcv

signature(object = "kha"): returns the principal component vectors

eskm

signature(object = "kha"): returns the kernel sum

predict

signature(object = "kha"): embeds new data

xmatrix

signature(object = "kha"): returns the used data matrix

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

kha, ksvm-class, kcca-class

Examples

# another example using the iris
data(iris)
test <- sample(1:50,20)

kpc <- kha(~.,data=iris[-test,-5], kernel="rbfdot",
           kpar=list(sigma=0.2),features=2, eta=0.001, maxiter=65)

#print the principal component vectors
pcv(kpc)
kernelf(kpc)
eig(kpc)

Description

A weighted kernel version of the famous k-means algorithm.

Usage

## S4 method for signature 'formula'
kkmeans(x, data = NULL, na.action = na.omit, ...)

## S4 method for signature 'matrix'
kkmeans(x, centers, kernel = "rbfdot", kpar = "automatic",
        alg="kkmeans", p=1, na.action = na.omit, ...)

## S4 method for signature 'kernelMatrix'
kkmeans(x, centers, ...)

## S4 method for signature 'list'
kkmeans(x, centers, kernel = "stringdot",
        kpar = list(length=4, lambda=0.5),
        alg ="kkmeans", p = 1, na.action = na.omit, ...)

Arguments

Details

kernel k-means uses the 'kernel trick' (i.e. implicitly projecting all data into a non-linear feature space with the use of a kernel) in order to deal with one of the major drawbacks of k-means that is that it cannot capture clusters that are not linearly separable in input space.
The algorithm is implemented using the triangle inequality to avoid unnecessary and computational expensive distance calculations. This leads to significant speedup particularly on large data sets with a high number of clusters.
With a particular choice of weights this algorithm becomes equivalent to Kernighan-Lin, and the norm-cut graph partitioning algorithms.
The function also support input in the form of a kernel matrix or a list of characters for text clustering.
The data can be passed to the kkmeans function in a matrix or a data.frame, in addition kkmeans also supports input in the form of a kernel matrix of class kernelMatrix or as a list of character vectors where a string kernel has to be used.

Value

An S4 object of class specc which extends the class vector containing integers indicating the cluster to which each point is allocated. The following slots contain useful information

Author(s)

Alexandros Karatzoglou
[email protected]

References

Inderjit Dhillon, Yuqiang Guan, Brian Kulis
A Unified view of Kernel k-means, Spectral Clustering and Graph Partitioning
UTCS Technical Report
https://people.bu.edu/bkulis/pubs/spectral_techreport.pdf

See Also

specc, kpca, kcca

Examples

## Cluster the iris data set.
data(iris)

sc <- kkmeans(as.matrix(iris[,-5]), centers=3)

sc
centers(sc)
size(sc)
withinss(sc)

Description

The Kernel Maximum Mean Discrepancy kmmd performs a non-parametric distribution test.

Usage

## S4 method for signature 'matrix'
kmmd(x, y, kernel="rbfdot",kpar="automatic", alpha = 0.05,
     asymptotic = FALSE, replace = TRUE, ntimes = 150, frac = 1, ...)

## S4 method for signature 'kernelMatrix'
kmmd(x, y, Kxy, alpha = 0.05,
     asymptotic = FALSE, replace = TRUE, ntimes = 100, frac = 1, ...)

## S4 method for signature 'list'
kmmd(x, y, kernel="stringdot", 
     kpar = list(type = "spectrum", length = 4), alpha = 0.05,
     asymptotic = FALSE, replace = TRUE, ntimes = 150, frac = 1, ...)

Arguments

Details

kmmd calculates the kernel maximum mean discrepancy for samples from two distributions and conducts a test as to whether the samples are from different distributions with level alpha.

Value

An S4 object of class kmmd containing the results of whether the H0 hypothesis is rejected or not. H0 being that the samples $x$ and $y$ come from the same distribution. The object contains the following slots :

see kmmd-class for more details.

Author(s)

Alexandros Karatzoglou
[email protected]

References

Gretton, A., K. Borgwardt, M. Rasch, B. Schoelkopf and A. Smola
A Kernel Method for the Two-Sample-Problem
Neural Information Processing Systems 2006, Vancouver
https://papers.neurips.cc/paper/3110-a-kernel-method-for-the-two-sample-problem.pdf

See Also

ksvm

Examples

# create data
x <- matrix(runif(300),100)
y <- matrix(runif(300)+1,100)


mmdo <- kmmd(x, y)

mmdo

Description

The Kernel Maximum Mean Discrepancy object class

Objects from the Class

Objects can be created by calls of the form new("kmmd", ...). or by calling the kmmd function

Slots

kernelf:

Object of class "kfunction" contains the kernel function used

xmatrix:

Object of class "kernelMatrix" containing the data used

H0

Object of class "logical" contains value of : is H0 rejected (logical)

AsympH0

Object of class "logical" contains value : is H0 rejected according to the asymptotic bound (logical)

mmdstats

Object of class "vector" contains the test statistics (vector of two)

Radbound

Object of class "numeric" contains the Rademacher bound

Asymbound

Object of class "numeric" contains the asymptotic bound

Methods

kernelf

signature(object = "kmmd"): returns the kernel function used

H0

signature(object = "kmmd"): returns the value of H0 being rejected

AsympH0

signature(object = "kmmd"): returns the value of H0 being rejected according to the asymptotic bound

mmdstats

signature(object = "kmmd"): returns the values of the mmd statistics

Radbound

signature(object = "kmmd"): returns the value of the Rademacher bound

Asymbound

signature(object = "kmmd"): returns the value of the asymptotic bound

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

kmmd,

Examples

# create data
x <- matrix(runif(300),100)
y <- matrix(runif(300)+1,100)


mmdo <- kmmd(x, y)

H0(mmdo)

Description

Kernel Principal Components Analysis is a nonlinear form of principal component analysis.

Usage

## S4 method for signature 'formula'
kpca(x, data = NULL, na.action, ...)

## S4 method for signature 'matrix'
kpca(x, kernel = "rbfdot", kpar = list(sigma = 0.1),
    features = 0, th = 1e-4, na.action = na.omit, ...)

## S4 method for signature 'kernelMatrix'
kpca(x, features = 0, th = 1e-4, ...)

## S4 method for signature 'list'
kpca(x, kernel = "stringdot", kpar = list(length = 4, lambda = 0.5),
    features = 0, th = 1e-4, na.action = na.omit, ...)

Arguments

Details

Using kernel functions one can efficiently compute principal components in high-dimensional feature spaces, related to input space by some non-linear map.
The data can be passed to the kpca function in a matrix or a data.frame, in addition kpca also supports input in the form of a kernel matrix of class kernelMatrix or as a list of character vectors where a string kernel has to be used.

Value

An S4 object containing the principal component vectors along with the corresponding eigenvalues.

all the slots of the object can be accessed by accessor functions.

Note

The predict function can be used to embed new data on the new space

Author(s)

Alexandros Karatzoglou
[email protected]

References

Schoelkopf B., A. Smola, K.-R. Mueller :
Nonlinear component analysis as a kernel eigenvalue problem
Neural Computation 10, 1299-1319
doi:10.1162/089976698300017467.

See Also

kcca, pca

Examples

# another example using the iris
data(iris)
test <- sample(1:150,20)

kpc <- kpca(~.,data=iris[-test,-5],kernel="rbfdot",
            kpar=list(sigma=0.2),features=2)

#print the principal component vectors
pcv(kpc)

#plot the data projection on the components
plot(rotated(kpc),col=as.integer(iris[-test,5]),
     xlab="1st Principal Component",ylab="2nd Principal Component")

#embed remaining points 
emb <- predict(kpc,iris[test,-5])
points(emb,col=as.integer(iris[test,5]))

Description

The Kernel Principal Components Analysis class

Objects of class "kpca"

Objects can be created by calls of the form new("kpca", ...). or by calling the kpca function.

Slots

pcv:

Object of class "matrix" containing the principal component vectors

eig:

Object of class "vector" containing the corresponding eigenvalues

rotated:

Object of class "matrix" containing the projection of the data on the principal components

kernelf:

Object of class "function" containing the kernel function used

kpar:

Object of class "list" containing the kernel parameters used

xmatrix:

Object of class "matrix" containing the data matrix used

kcall:

Object of class "ANY" containing the function call

n.action:

Object of class "ANY" containing the action performed on NA

Methods

eig

signature(object = "kpca"): returns the eigenvalues

kcall

signature(object = "kpca"): returns the performed call

kernelf

signature(object = "kpca"): returns the used kernel function

pcv

signature(object = "kpca"): returns the principal component vectors

predict

signature(object = "kpca"): embeds new data

rotated

signature(object = "kpca"): returns the projected data

xmatrix

signature(object = "kpca"): returns the used data matrix

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

ksvm-class, kcca-class

Examples

# another example using the iris
data(iris)
test <- sample(1:50,20)

kpc <- kpca(~.,data=iris[-test,-5],kernel="rbfdot",
            kpar=list(sigma=0.2),features=2)

#print the principal component vectors
pcv(kpc)
rotated(kpc)
kernelf(kpc)
eig(kpc)

Description

The Kernel Quantile Regression algorithm kqr performs non-parametric Quantile Regression.

Usage

## S4 method for signature 'formula'
kqr(x, data=NULL, ..., subset, na.action = na.omit, scaled = TRUE)

## S4 method for signature 'vector'
kqr(x,...)

## S4 method for signature 'matrix'
kqr(x, y, scaled = TRUE, tau = 0.5, C = 0.1, kernel = "rbfdot",
    kpar = "automatic", reduced = FALSE, rank = dim(x)[1]/6,
    fit = TRUE, cross = 0, na.action = na.omit)

## S4 method for signature 'kernelMatrix'
kqr(x, y, tau = 0.5, C = 0.1, fit = TRUE, cross = 0)

## S4 method for signature 'list'
kqr(x, y, tau = 0.5, C = 0.1, kernel = "strigdot",
    kpar= list(length=4, C=0.5), fit = TRUE, cross = 0)

Arguments

Details

In quantile regression a function is fitted to the data so that it satisfies the property that a portion $tau$ of the data $y|n$ is below the estimate. While the error bars of many regression problems can be viewed as such estimates quantile regression estimates this quantity directly. Kernel quantile regression is similar to nu-Support Vector Regression in that it minimizes a regularized loss function in RKHS. The difference between nu-SVR and kernel quantile regression is in the type of loss function used which in the case of quantile regression is the pinball loss (see reference for details.). Minimizing the regularized loss boils down to a quadratic problem which is solved using an interior point QP solver ipop implemented in kernlab.

Value

An S4 object of class kqr containing the fitted model along with information.Accessor functions can be used to access the slots of the object which include :

see kqr-class for more details.

Author(s)

Alexandros Karatzoglou
[email protected]

References

Ichiro Takeuchi, Quoc V. Le, Timothy D. Sears, Alexander J. Smola
Nonparametric Quantile Estimation
Journal of Machine Learning Research 7,2006,1231-1264
https://www.jmlr.org/papers/volume7/takeuchi06a/takeuchi06a.pdf

See Also

predict.kqr, kqr-class, ipop, rvm, ksvm

Examples

# create data
x <- sort(runif(300))
y <- sin(pi*x) + rnorm(300,0,sd=exp(sin(2*pi*x)))

# first calculate the median
qrm <- kqr(x, y, tau = 0.5, C=0.15)

# predict and plot
plot(x, y)
ytest <- predict(qrm, x)
lines(x, ytest, col="blue")

# calculate 0.9 quantile
qrm <- kqr(x, y, tau = 0.9, kernel = "rbfdot",
           kpar= list(sigma=10), C=0.15)
ytest <- predict(qrm, x)
lines(x, ytest, col="red")

# calculate 0.1 quantile
qrm <- kqr(x, y, tau = 0.1,C=0.15)
ytest <- predict(qrm, x)
lines(x, ytest, col="green")

# print first 10 model coefficients
coef(qrm)[1:10]

Description

The Kernel Quantile Regression object class

Objects from the Class

Objects can be created by calls of the form new("kqr", ...). or by calling the kqr function

Slots

kernelf:

Object of class "kfunction" contains the kernel function used

kpar:

Object of class "list" contains the kernel parameter used

coef:

Object of class "ANY" containing the model parameters

param:

Object of class "list" contains the cost parameter C and tau parameter used

kcall:

Object of class "list" contains the used function call

terms:

Object of class "ANY" contains the terms representation of the symbolic model used (when using a formula)

xmatrix:

Object of class "input" containing the data matrix used

ymatrix:

Object of class "output" containing the response matrix

fitted:

Object of class "output" containing the fitted values

alpha:

Object of class "listI" containing the computes alpha values

b:

Object of class "numeric" containing the offset of the model.

scaling

Object of class "ANY" containing the scaling coefficients of the data (when case scaled = TRUE is used).

error:

Object of class "numeric" containing the training error

cross:

Object of class "numeric" containing the cross validation error

n.action:

Object of class "ANY" containing the action performed in NA

nclass:

Inherited from class vm, not used in kqr

lev:

Inherited from class vm, not used in kqr

type:

Inherited from class vm, not used in kqr

Methods

coef

signature(object = "kqr"): returns the coefficients (alpha) of the model

alpha

signature(object = "kqr"): returns the alpha vector (identical to coef)

b

signature(object = "kqr"): returns the offset beta of the model.

cross

signature(object = "kqr"): returns the cross validation error

error

signature(object = "kqr"): returns the training error

fitted

signature(object = "vm"): returns the fitted values

kcall

signature(object = "kqr"): returns the call performed

kernelf

signature(object = "kqr"): returns the kernel function used

kpar

signature(object = "kqr"): returns the kernel parameter used

param

signature(object = "kqr"): returns the cost regularization parameter C and tau used

xmatrix

signature(object = "kqr"): returns the data matrix used

ymatrix

signature(object = "kqr"): returns the response matrix used

scaling

signature(object = "kqr"): returns the scaling coefficients of the data (when scaled = TRUE is used)

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

kqr, vm-class, ksvm-class

Examples

# create data
x <- sort(runif(300))
y <- sin(pi*x) + rnorm(300,0,sd=exp(sin(2*pi*x)))

# first calculate the median
qrm <- kqr(x, y, tau = 0.5, C=0.15)

# predict and plot
plot(x, y)
ytest <- predict(qrm, x)
lines(x, ytest, col="blue")

# calculate 0.9 quantile
qrm <- kqr(x, y, tau = 0.9, kernel = "rbfdot",
           kpar = list(sigma = 10), C = 0.15)
ytest <- predict(qrm, x)
lines(x, ytest, col="red")

# print model coefficients and other information
coef(qrm)
b(qrm)
error(qrm)
kernelf(qrm)

Description

Support Vector Machines are an excellent tool for classification, novelty detection, and regression. ksvm supports the well known C-svc, nu-svc, (classification) one-class-svc (novelty) eps-svr, nu-svr (regression) formulations along with native multi-class classification formulations and the bound-constraint SVM formulations.
ksvm also supports class-probabilities output and confidence intervals for regression.

Usage

## S4 method for signature 'formula'
ksvm(x, data = NULL, ..., subset, na.action = na.omit, scaled = TRUE)

## S4 method for signature 'vector'
ksvm(x, ...)

## S4 method for signature 'matrix'
ksvm(x, y = NULL, scaled = TRUE, type = NULL,
     kernel ="rbfdot", kpar = "automatic",
     C = 1, nu = 0.2, epsilon = 0.1, prob.model = FALSE,
     class.weights = NULL, cross = 0, fit = TRUE, cache = 40,
     tol = 0.001, shrinking = TRUE, ..., 
     subset, na.action = na.omit)

## S4 method for signature 'kernelMatrix'
ksvm(x, y = NULL, type = NULL,
     C = 1, nu = 0.2, epsilon = 0.1, prob.model = FALSE,
     class.weights = NULL, cross = 0, fit = TRUE, cache = 40,
     tol = 0.001, shrinking = TRUE, ...)

## S4 method for signature 'list'
ksvm(x, y = NULL, type = NULL,
     kernel = "stringdot", kpar = list(length = 4, lambda = 0.5),
     C = 1, nu = 0.2, epsilon = 0.1, prob.model = FALSE,
     class.weights = NULL, cross = 0, fit = TRUE, cache = 40,
     tol = 0.001, shrinking = TRUE, ...,
     na.action = na.omit)

Arguments

Details

ksvm uses John Platt's SMO algorithm for solving the SVM QP problem an most SVM formulations. On the spoc-svc, kbb-svc, C-bsvc and eps-bsvr formulations a chunking algorithm based on the TRON QP solver is used.
For multiclass-classification with $k$ classes, $k > 2$, ksvm uses the ‘one-against-one’-approach, in which $k(k-1)/2$ binary classifiers are trained; the appropriate class is found by a voting scheme, The spoc-svc and the kbb-svc formulations deal with the multiclass-classification problems by solving a single quadratic problem involving all the classes.
If the predictor variables include factors, the formula interface must be used to get a correct model matrix.
In classification when prob.model is TRUE a 3-fold cross validation is performed on the data and a sigmoid function is fitted on the resulting decision values $f$. The data can be passed to the ksvm function in a matrix or a data.frame, in addition ksvm also supports input in the form of a kernel matrix of class kernelMatrix or as a list of character vectors where a string kernel has to be used.
The plot function for binary classification ksvm objects displays a contour plot of the decision values with the corresponding support vectors highlighted.
The predict function can return class probabilities for classification problems by setting the type parameter to "probabilities".
The problem of model selection is partially addressed by an empirical observation for the RBF kernels (Gaussian , Laplace) where the optimal values of the $sigma$ width parameter are shown to lie in between the 0.1 and 0.9 quantile of the $\|x- x'\|$ statistics. When using an RBF kernel and setting kpar to "automatic", ksvm uses the sigest function to estimate the quantiles and uses the median of the values.

Value

An S4 object of class "ksvm" containing the fitted model, Accessor functions can be used to access the slots of the object (see examples) which include:

Note

Data is scaled internally by default, usually yielding better results.

Author(s)

Alexandros Karatzoglou (SMO optimizers in C++ by Chih-Chung Chang & Chih-Jen Lin)
[email protected]

References

• Chang Chih-Chung, Lin Chih-Jen
  LIBSVM: a library for Support Vector Machines
  https://www.csie.ntu.edu.tw/~cjlin/libsvm/

• Chih-Wei Hsu, Chih-Jen Lin
  BSVM https://www.csie.ntu.edu.tw/~cjlin/bsvm/

• J. Platt
  Probabilistic outputs for support vector machines and comparison to regularized likelihood methods
  Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Schoelkopf and D. Schuurmans, Eds. Cambridge, MA: MIT Press, 2000.

• H.-T. Lin, C.-J. Lin and R. C. Weng
  A note on Platt's probabilistic outputs for support vector machines
  https://www.csie.ntu.edu.tw/~htlin/paper/doc/plattprob.pdf

• C.-W. Hsu and C.-J. Lin
  A comparison on methods for multi-class support vector machines
  IEEE Transactions on Neural Networks, 13(2002) 415-425.
  https://www.csie.ntu.edu.tw/~cjlin/papers/multisvm.pdf

• K. Crammer, Y. Singer
  On the learnability and design of output codes for multiclass prolems
  Computational Learning Theory, 35-46, 2000.
  http://www.learningtheory.org/colt2000/papers/CrammerSinger.pdf

• J. Weston, C. Watkins
  Multi-class support vector machines. Technical Report CSD-TR-98-04, Royal Holloway, University of London, Department of Computer Science.

See Also

predict.ksvm, ksvm-class, couple

Examples

## simple example using the spam data set
data(spam)

## create test and training set
index <- sample(1:dim(spam)[1])
spamtrain <- spam[index[1:floor(dim(spam)[1]/2)], ]
spamtest <- spam[index[((ceiling(dim(spam)[1]/2)) + 1):dim(spam)[1]], ]

## train a support vector machine
filter <- ksvm(type~.,data=spamtrain,kernel="rbfdot",
               kpar=list(sigma=0.05),C=5,cross=3)
filter

## predict mail type on the test set
mailtype <- predict(filter,spamtest[,-58])

## Check results
table(mailtype,spamtest[,58])


## Another example with the famous iris data
data(iris)

## Create a kernel function using the build in rbfdot function
rbf <- rbfdot(sigma=0.1)
rbf

## train a bound constraint support vector machine
irismodel <- ksvm(Species~.,data=iris,type="C-bsvc",
                  kernel=rbf,C=10,prob.model=TRUE)

irismodel

## get fitted values
fitted(irismodel)

## Test on the training set with probabilities as output
predict(irismodel, iris[,-5], type="probabilities")


## Demo of the plot function
x <- rbind(matrix(rnorm(120),,2),matrix(rnorm(120,mean=3),,2))
y <- matrix(c(rep(1,60),rep(-1,60)))

svp <- ksvm(x,y,type="C-svc")
plot(svp,data=x)


### Use kernelMatrix
K <- as.kernelMatrix(crossprod(t(x)))

svp2 <- ksvm(K, y, type="C-svc")

svp2

# test data
xtest <- rbind(matrix(rnorm(20),,2),matrix(rnorm(20,mean=3),,2))
# test kernel matrix i.e. inner/kernel product of test data with
# Support Vectors

Ktest <- as.kernelMatrix(crossprod(t(xtest),t(x[SVindex(svp2), ])))

predict(svp2, Ktest)


#### Use custom kernel 

k <- function(x,y) {(sum(x*y) +1)*exp(-0.001*sum((x-y)^2))}
class(k) <- "kernel"

data(promotergene)

## train svm using custom kernel
gene <- ksvm(Class~.,data=promotergene[c(1:20, 80:100),],kernel=k,
             C=5,cross=5)

gene


#### Use text with string kernels
data(reuters)
is(reuters)
tsv <- ksvm(reuters,rlabels,kernel="stringdot",
            kpar=list(length=5),cross=3,C=10)
tsv


## regression
# create data
x <- seq(-20,20,0.1)
y <- sin(x)/x + rnorm(401,sd=0.03)

# train support vector machine
regm <- ksvm(x,y,epsilon=0.01,kpar=list(sigma=16),cross=3)
plot(x,y,type="l")
lines(x,predict(regm,x),col="red")

Description

An S4 class containing the output (model) of the ksvm Support Vector Machines function

Objects from the Class

Objects can be created by calls of the form new("ksvm", ...) or by calls to the ksvm function.

Slots

type:

Object of class "character" containing the support vector machine type ("C-svc", "nu-svc", "C-bsvc", "spoc-svc", "one-svc", "eps-svr", "nu-svr", "eps-bsvr")

param:

Object of class "list" containing the Support Vector Machine parameters (C, nu, epsilon)

kernelf:

Object of class "function" containing the kernel function

kpar:

Object of class "list" containing the kernel function parameters (hyperparameters)

kcall:

Object of class "ANY" containing the ksvm function call

scaling:

Object of class "ANY" containing the scaling information performed on the data

terms:

Object of class "ANY" containing the terms representation of the symbolic model used (when using a formula)

xmatrix:

Object of class "input" ("list" for multiclass problems or "matrix" for binary classification and regression problems) containing the support vectors calculated from the data matrix used during computations (possibly scaled and without NA). In the case of multi-class classification each list entry contains the support vectors from each binary classification problem from the one-against-one method.

ymatrix:

Object of class "output" the response "matrix" or "factor" or "vector" or "logical"

fitted:

Object of class "output" with the fitted values, predictions using the training set.

lev:

Object of class "vector" with the levels of the response (in the case of classification)

prob.model:

Object of class "list" with the class prob. model

prior:

Object of class "list" with the prior of the training set

nclass:

Object of class "numeric" containing the number of classes (in the case of classification)

alpha:

Object of class "listI" containing the resulting alpha vector ("list" or "matrix" in case of multiclass classification) (support vectors)

coef:

Object of class "ANY" containing the resulting coefficients

alphaindex:

Object of class "list" containing

b:

Object of class "numeric" containing the resulting offset

SVindex:

Object of class "vector" containing the indexes of the support vectors

nSV:

Object of class "numeric" containing the number of support vectors

obj:

Object of class vector containing the value of the objective function. When using one-against-one in multiclass classification this is a vector.

error:

Object of class "numeric" containing the training error

cross:

Object of class "numeric" containing the cross-validation error

n.action:

Object of class "ANY" containing the action performed for NA

Methods

SVindex

signature(object = "ksvm"): return the indexes of support vectors

alpha

signature(object = "ksvm"): returns the complete 5 alpha vector (wit zero values)

alphaindex

signature(object = "ksvm"): returns the indexes of non-zero alphas (support vectors)

cross

signature(object = "ksvm"): returns the cross-validation error

error

signature(object = "ksvm"): returns the training error

obj

signature(object = "ksvm"): returns the value of the objective function

fitted

signature(object = "vm"): returns the fitted values (predict on training set)

kernelf

signature(object = "ksvm"): returns the kernel function

kpar

signature(object = "ksvm"): returns the kernel parameters (hyperparameters)

lev

signature(object = "ksvm"): returns the levels in case of classification

prob.model

signature(object="ksvm"): returns class prob. model values

param

signature(object="ksvm"): returns the parameters of the SVM in a list (C, epsilon, nu etc.)

prior

signature(object="ksvm"): returns the prior of the training set

kcall

signature(object="ksvm"): returns the ksvm function call

scaling

signature(object = "ksvm"): returns the scaling values

show

signature(object = "ksvm"): prints the object information

type

signature(object = "ksvm"): returns the problem type

xmatrix

signature(object = "ksvm"): returns the data matrix used

ymatrix

signature(object = "ksvm"): returns the response vector

Author(s)

Alexandros Karatzoglou
[email protected]

See Also

ksvm, rvm-class, gausspr-class

Examples

## simple example using the promotergene data set
data(promotergene)

## train a support vector machine
gene <- ksvm(Class~.,data=promotergene,kernel="rbfdot",
             kpar=list(sigma=0.015),C=50,cross=4)
gene

# the kernel  function
kernelf(gene)
# the alpha values
alpha(gene)
# the coefficients
coef(gene)
# the fitted values
fitted(gene)
# the cross validation error
cross(gene)