Developer Guide and Reference

  • 2021.2
  • 03/26/2021
  • Public Content
Contents

Least Absolute Shrinkage and Selection Operator (LASSO)

Least Absolute Shrinkage and Selection Operator (LASSO) is a method for modeling relationship between a dependent variable (which may be a vector) and one or more explanatory variables by fitting regularized least squares model. Trained LASSO model can produce sparse coefficients due to the use of LaTex Math image. regularization term. LASSO regression is widely used in feature selection tasks. For example, in the field of compressed sensing it is used to effectively identify relevant features associated with the dependent variable from a few observations with a large number of features. LASSO regression is also used to overcome multicollinearity of feature vectors in the training data set.

Details

Let LaTex Math image. be a vector of input variables and LaTex Math image. be the response. For each
j = 1, …, k
, the LASSO model has the form similar to linear and ridge regression model [Hoerl70], except that the coefficients are trained by minimizing a regularized by LaTex Math image. penalty mean squared error (MSE) objective function.
LaTex Math image.
Here LaTex Math image.,
i = 1, ldots, p
are referred to as independent variables, LaTex Math image. is referred to as dependent variable or response and
j = 1, …, k
.
Training Stage
Let LaTex Math image. be a set of training data (for regression task, LaTex Math image., and for feature selection
p
could be greater than
n
). The matrix
X
of size LaTex Math image. contains observations LaTex Math image.,
i = 1, …, n
,
j = 1, ldots, p
of independent variables.
For each LaTex Math image.,
j = 1, …, k
, the LASSO regression estimates LaTex Math image. by minimizing the objective function:
LaTex Math image.
In the equation above, the first term is a mean squared error function and the second one is a regularization term that penalizes the LaTex Math image. norm of vector LaTex Math image.
For more details, see [Hastie2009].
By default, Coordinate Descent iterative solver is used to minimize the objective function. SAGA solver is also applicable for minimization.
Prediction Stage
For input vector of independent variables LaTex Math image., prediction based on LASSO regression is done using the equation
LaTex Math image.
where
j = 1, …, k
.

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.