Developer Guide and Reference

  • 2021.3
  • 06/28/2021
  • Public Content
Contents

Stochastic Gradient Descent Algorithm

The stochastic gradient descent (SGD) algorithm is a special case of an iterative solver. See Iterative Solver for more details.

Computation methods

The following computation methods are available in oneDAL for the stochastic gradient descent algorithm:
Mini-batch method
The mini-batch method (miniBatch) of the stochastic gradient descent algorithm [Mu2014] follows the algorithmic framework of an iterative solver with an empty set of intrinsic parameters of the algorithm LaTex Math image., algorithm-specific transformation
T
defined for the learning rate sequence LaTex Math image., conservative sequence LaTex Math image. and the number of iterations in the internal loop
L
, algorithm-specific vector
U
and power
d
of Lebesgue space defined as follows:
LaTex Math image.
For
l
from
1
until
L
:
  1. Update the function argument: LaTex Math image.
  2. Compute the gradient: LaTex Math image.
Convergence check: LaTex Math image.
Default method
The default method (defaultDense) is a particular case of the mini-batch method with the batch size LaTex Math image., LaTex Math image., and conservative sequence LaTex Math image..
Momentum method
The momentum method (momentum) of the stochastic gradient descent algorithm [Rumelhart86] follows the algorithmic framework of an iterative solver with the set of intrinsic parameters LaTex Math image., algorithm-specific transformation
T
defined for the learning rate sequence LaTex Math image. and momentum parameter LaTex Math image., and algorithm-specific vector
U
and power
d
of Lebesgue space defined as follows:
LaTex Math image.
  1. LaTex Math image.
  2. LaTex Math image.
For the momentum method of the SGD algorithm, the set of intrinsic parameters LaTex Math image. only contains the last update vector LaTex Math image..
Convergence check: LaTex Math image.

Computation

The stochastic gradient descent algorithm is a special case of an iterative solver. For parameters, input, and output of iterative solvers, see Computation.
Algorithm Parameters
In addition to parameters of the iterative solver, the stochastic gradient descent algorithm has the following parameters. Some of them are required only for specific values of the computation method parameter method:
Parameter
method
Default Value
Description
algorithmFPType
defaultDense
,
miniBatch
,
momentum
float
The floating-point type that the algorithm uses for intermediate computations. Can be
float
or
double
.
method
Not applicable
defaultDense
Available computation methods:
For CPU:
  • defaultDense
  • miniBatch
  • momentum
For GPU:
  • miniBatch
batchIndices
defaultDense
,
miniBatch
,
momentum
Not applicable
The numeric table with 32-bit integer indices of terms in the objective function. The method parameter determines the size of the numeric table:
  • defaultDense
    :
    nIterations
    x 1
  • miniBatch
    and
    momentum
    :
    nIterations
    x
    batchSize
If no indices are provided, the implementation generates random indices.
batchSize
miniBatch
,``momentum``
128
The number of batch indices to compute the stochastic gradient.
If
batchSize
equals the number of terms in the objective function, no random sampling is performed, and all terms are used to calculate the gradient.
The algorithm ignores this parameter if the batchIndices parameter is provided.
For the
defaultDense
value of method, one term is used to compute the gradient on each iteration.
conservativeSequence
miniBatch
A numeric table of size LaTex Math image. that contains the default conservative coefficient equal to 1.
The numeric table of size LaTex Math image. or LaTex Math image.. The contents of the table depend on its size:
  • size = LaTex Math image.: values of the conservative coefficient sequence LaTex Math image. for LaTex Math image..
  • size = LaTex Math image. the value of conservative coefficient at each iteration LaTex Math image..
innerNIterations
miniBatch
5
The number of inner iterations for the miniBatch method.
learningRateSequence
defaultDense
,
miniBatch
,
momentum
A numeric table of size LaTex Math image. that contains the default step length equal to 1.
The numeric table of size LaTex Math image. or LaTex Math image.. The contents of the table depend on its size:
  • size = LaTex Math image.: values of the learning rate sequence LaTex Math image. for LaTex Math image..
  • size = LaTex Math image.: the value of learning rate at each iteration LaTex Math image..
momentum
momentum
0.9
The momentum value.
engine
defaultDense
,
miniBatch
,
momentum
SharePtr< engines:: mt19937:: Batch>()
Pointer to the random number generator engine that is used internally for generation of 32-bit integer indices of terms in the objective function.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.