Stochastic Average Gradient Accelerated Method

The Stochastic Average Gradient Accelerated (SAGA) [Defazio2014] follows the algorithmic framework of an iterative solver with one exception.

The default method (defaultDense) of SAGA algorithm is a particular case of the iterative solver method with the batch size b=1.

Algorithmic-specific transformation T, set of intrinsic parameters S t defined for the learning η rate, and algorithm-specific vector U and power d of Lebesgue space are defined as follows:

S t - a matrix of the gradients of smooth terms at point , where

  • t is defined by the number of iterations the solver runs

  • stores a gradient of

:

Update of the set of intrinsic parameters S t: .

Note

The algorithm enables automatic step-length selection if learning rate η was not provided by the user. Automatic step-length will be computed as , where L - Lipschitz constant returned by objective function. If the objective function returns nullptr to numeric table with lipschitzConstantresult-id, the library will use default step size '0.01'.

Convergence check:

  • ;

For more complete information about compiler optimizations, see our Optimization Notice.
Select sticky button color: 
Orange (only for download buttons)