Computation

The stochastic gradient descent algorithm is a special case of an iterative solver. For parameters, input, and output of iterative solvers, see Iterative Solver > Computation.

Algorithm Parameters

In addition to parameters of the iterative solver, the stochastic gradient descent algorithm has the following parameters. Some of them are required only for specific values of the computation method parameter method:

Parameter

method

Default Value

Description

algorithmFPType

defaultDense,

miniBatch,

or

momentum

float

The floating-point type that the algorithm uses for intermediate computations. Can be float or double.

method

Not applicable

defaultDense

Available computation methods:

  • defaultDense
  • miniBatch
  • momentum

batchIndices

defaultDense,

miniBatch,

or

momentum

NULL

The numeric table with 32-bit integer indices of terms in the objective function. The method parameter determines the size of the numeric table:

  • defaultDense: nIterations x 1
  • miniBatch or momentum: nIterations x batchSize

If no indices are provided, the implementation generates random indices.

This parameter can be an object of any class derived from NumericTable except PackedTriangularMatrix, PackedSymmetricMatrix, and CSRNumericTable.

batchSize

miniBatch

or

momentum

128

Number of batch indices to compute the stochastic gradient. If batchSize equals the number of terms in the objective function, no random sampling is performed, and all terms are used to calculate the gradient.

The algorithm ignores this parameter if the batchIndices parameter is provided.

For the defaultDense value of method, one term is used to compute the gradient on each iteration.

conservativeSequence

miniBatch

Numeric table of size 1 x 1 that contains the default conservative coefficient equal to 1.

The numeric table of size 1 x nIterations or 1 x 1. The contents of the table depend on its size:

  • size = 1 x nIterations: values of the conservative coefficient sequence γk for k = 1, …, nIterations.
  • size = 1 x 1: the value of conservative coefficient at each iteration γ1 = ... = γnIterations.

This parameter can be an object of any class derived from NumericTable except PackedTriangularMatrix, PackedSymmetricMatrix, and CSRNumericTable.

innerNIterations

miniBatch

5

The number of inner iterations for the miniBatch method.

learningRateSequence

defaultDense,

miniBatch,

or

momentum

Numeric table of size 1 x 1 that contains the default step length equal to 1.

The numeric table of size 1 x nIterations or 1 x 1. The contents of the table depend on its size:

  • size = 1 x nIterations: values of the learning rate sequence ηk for k = 1, …, nIterations.
  • size = 1 x 1: the value of learning rate at each iteration η1 = ... = ηnIterations.

This parameter can be an object of any class derived from NumericTable except PackedTriangularMatrix, PackedSymmetricMatrix, and CSRNumericTable.

momentum

momentum

0.9

The momentum value.

engine

defaultDense,

miniBatch,

or

momentum

SharePtr< engines:: mt19937:: Batch>()

Pointer to the random number generator engine that is used internally for generation of 32-bit integer indices of terms in the objective function.

For more complete information about compiler optimizations, see our Optimization Notice.
Select sticky button color: 
Orange (only for download buttons)