Developer Guide and Reference

  • 2021.3
  • 06/28/2021
  • Public Content
Contents

Support Vector Machine Classifier and Regression (SVM)

Support Vector Machine (SVM) classification and regression are among popular algorithms. It belongs to a family of generalized linear classification problems.

Mathematical formulation

Training
Given
n
feature vectors LaTex Math image. of size
p
, their non-negative observation weights LaTex Math image., and
n
responses LaTex Math image.,
Classification
  • LaTex Math image., where
    M
    is the number of classes
Regression
  • LaTex Math image.
Nu-classification
  • LaTex Math image., where
    M
    is the number of classes
Nu-regression
  • LaTex Math image.
the problem is to build a Support Vector Machine (SVM) classification, regression, nu-classification, or nu-regression model.
The SVM model is trained using the Sequential minimal optimization (SMO) method [Boser92] for reduced to the solution of the quadratic optimization problem
Classification
LaTex Math image.
with LaTex Math image., LaTex Math image., LaTex Math image., where
e
is the vector of ones,
C
is the upper bound of the coordinates of the vector LaTex Math image.,
Q
is a symmetric matrix of size LaTex Math image. with LaTex Math image., and LaTex Math image. is a kernel function.
Regression
LaTex Math image.
with LaTex Math image., LaTex Math image., LaTex Math image., where
C
is the upper bound of the coordinates of the vector LaTex Math image.,
Q
is a symmetric matrix of size LaTex Math image. with LaTex Math image., and LaTex Math image. is a kernel function. Vectors
s
and
z
for the regression problem are formulated according to the following rule:
LaTex Math image.
Where LaTex Math image. is the error tolerance parameter.
Nu-classification
LaTex Math image.
with LaTex Math image., LaTex Math image., LaTex Math image., LaTex Math image., where
e
is the vector of ones, LaTex Math image. is an upper bound on the fraction of training errors and a lower bound of the fraction of the support vector,
Q
is a symmetric matrix of size LaTex Math image. with LaTex Math image., and LaTex Math image. is a kernel function.
Nu-regression
LaTex Math image.
with LaTex Math image., LaTex Math image., LaTex Math image., LaTex Math image., where
C
is the upper bound of the coordinates of the vector LaTex Math image., LaTex Math image. is an upper bound on the fraction of training errors and a lower bound of the fraction of the support vector,
Q
is a symmetric matrix of size LaTex Math image. with LaTex Math image., and LaTex Math image. is a kernel function. Vector
z
for the regression problem are formulated according to the following rule:
LaTex Math image.
Working subset of α updated on each iteration of the algorithm is based on the Working Set Selection (WSS) 3 scheme [Fan05]. The scheme can be optimized using one of these techniques or both:
  • Cache
    : the implementation can allocate a predefined amount of memory to store intermediate results of the kernel computation.
  • Shrinking
    : the implementation can try to decrease the amount of kernel related computations (see [Joachims99]).
The solution of the problem defines the separating hyperplane and corresponding decision function LaTex Math image., where only those LaTex Math image. that correspond to non-zero LaTex Math image. appear in the sum, and
b
is a bias. Each non-zero LaTex Math image. is called a dual coefficient and the corresponding LaTex Math image. is called a support vector.
Training method:
smo
In
smo
training method, all vectors from the training dataset are used for each iteration.
Training method:
thunder
In
thunder
training method, the algorithm iteratively solves the convex optimization problem with the linear constraints by selecting the fixed set of active constrains (working set) and applying Sequential Minimal Optimization (SMO) solver to the selected subproblem. The description of this method is given in Algorithm [Wen2018].
Inference methods:
smo
and
thunder
smo
and
thunder
inference methods perform prediction in the same way:
Given the SVM classification or regression model and
r
feature vectors LaTex Math image., the problem is to calculate the signed value of the decision function LaTex Math image., LaTex Math image.. The sign of the value defines the class of the feature vector, and the absolute value of the function is a multiple of the distance between the feature vector and the separating hyperplane.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.