Developer Guide

Contents

Computation

The stochastic gradient descent algorithm is a special case of an iterative solver. For parameters, input, and output of iterative solvers, see Iterative Solver > Computation.

Algorithm Parameters

In addition to parameters of the iterative solver, the stochastic gradient descent algorithm has the following parameters. Some of them are required only for specific values of the computation method parameter
method
:
Parameter
method
Default Value
Description
algorithmFPType
defaultDense
,
miniBatch
,
or
momentum
float
The floating-point type that the algorithm uses for intermediate computations. Can be
float
or
double
.
method
Not applicable
defaultDense
Available computation methods:
  • defaultDense
  • miniBatch
  • momentum
batchIndices
defaultDense
,
miniBatch
,
or
momentum
NULL
The numeric table with 32-bit integer indices of terms in the objective function. The
method
parameter determines the size of the numeric table:
  • defaultDense:
    nIterations
    x 1
  • miniBatch
    or
    momentum:
    nIterations
    x
    batchSize
If no indices are provided, the implementation generates random indices.
This parameter can be an object of any class derived from
NumericTable
except
PackedTriangularMatrix
,
PackedSymmetricMatrix
, and
CSRNumericTable
.
batchSize
miniBatch
or
momentum
128
Number of batch indices to compute the stochastic gradient. If
batchSize
equals the number of terms in the objective function, no random sampling is performed, and all terms are used to calculate the gradient.
The algorithm ignores this parameter if the
batchIndices
parameter is provided.
For the
defaultDense
value of
method
, one term is used to compute the gradient on each iteration.
conservativeSequence
miniBatch
Numeric table of size 1 x 1 that contains the default conservative coefficient equal to 1.
The numeric table of size 1 x
nIterations
or 1 x 1. The contents of the table depend on its size:
  • size = 1 x
    nIterations
    : values of the conservative coefficient sequence γ
    k
    for
    k
    = 1, …,
    nIterations
    .
  • size = 1 x 1: the value of conservative coefficient at each iteration γ
    1
    = ... = γ
    nIterations
    .
This parameter can be an object of any class derived from
NumericTable
except
PackedTriangularMatrix
,
PackedSymmetricMatrix
, and
CSRNumericTable
.
innerNIterations
miniBatch
5
The number of inner iterations for the
miniBatch
method.
learningRateSequence
defaultDense
,
miniBatch
,
or
momentum
Numeric table of size 1 x 1 that contains the default step length equal to 1.
The numeric table of size 1 x
nIterations
or 1 x 1. The contents of the table depend on its size:
  • size = 1 x
    nIterations
    : values of the learning rate sequence η
    k
    for
    k
    = 1, …,
    nIterations
    .
  • size = 1 x 1: the value of learning rate at each iteration η
    1
    = ... = η
    nIterations
    .
This parameter can be an object of any class derived from
NumericTable
except
PackedTriangularMatrix
,
PackedSymmetricMatrix
, and
CSRNumericTable
.
momentum
momentum
0.9
The momentum value.
engine
defaultDense
,
miniBatch
,
or
momentum
SharePtr< engines:: mt19937:: Batch>()
Pointer to the random number generator engine that is used internally for generation of 32-bit integer indices of terms in the objective function.

Product and Performance Information

1

Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserverd for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.

Notice revision #20110804