Solvers

In general MLJLinearModels tries to use "reasonable defaults" for solvers. You may want to pick something else particularly if your data is extreme in some way (e.g. very noisy, or very large).

Only some solvers are appropriate for some models (see models) for a list.

MLJLinearModels.Analytical — Type

Analytical solver (Cholesky). If the iterative parameter is set to true then a CG solver is used. The CG solver is matrix-free and should be preferred in "large scale" cases (when the hat matrix X'X is "big").

Parameters

iterative (Bool): whether to use CG (iterative) or not
max_inner (Int): in the iterative mode, how many inner iterations to do.

source

MLJLinearModels.Newton — Type

Newton solver. This is a full Hessian solver and should be avoided for "large scale" cases.

optim_options are the general Optim Options. newton_options are the options of Newton's method

Example

using MLJLinearModels, Optim

solver = MLJLinearModels.Newton(
    optim_options = Optim.Options(time_limit = 20),
    newton_options = (linesearch = Optim.LineSearches.HagerZhang()),)
)

source

MLJLinearModels.NewtonCG — Type

Newton CG solver. This is the same as the Newton solver except that instead of solving systems of the form H\b where H is the full Hessian, it uses a matrix-free conjugate gradient approach to solving that system. This should generally be preferred for larger scale cases.

optim_options are the general Optim Options. newtoncg_options are the options of Krylov Trust Region method

Example

using MLJLinearModels, Optim

solver = MLJLinearModels.NewtonCG(
    optim_options = Optim.Options(time_limit = 20),
    newtoncg_options = (eta = 0.2,)
)

source

MLJLinearModels.LBFGS — Type

LBFGS quasi-Newton solver. See the wikipedia entry.

optim_options are the general Optim Options. lbfgs_options are the options of LBFGS method

Example

using MLJLinearModels, Optim

solver = MLJLinearModels.LBFGS(
    optim_options = Optim.Options(time_limit = 20),
    lbfgs_options = (linesearch = Optim.LineSearches.HagerZhang()),)
)

source

MLJLinearModels.ProxGrad — Type

Proximal Gradient solver for non-smooth objective functions.

Parameters

accel (Bool): whether to use Nesterov-style acceleration
max_iter (Int): number of overall iterations
tol (Float64): tolerance for the relative change θ ie norm(θ-θ_)/norm(θ)
max_inner: number of inner steps when searching for a stepsize in the backtracking step
beta: rate of shrinkage in the backtracking step (between 0 and 1)

source

MLJLinearModels.IWLSCG — Type

Iteratively Reweighted Least Square with Conjugate Gradient. This is the standard (expensive) IWLS but with more efficient solves to avoid full matrix computations.

Parameters

max_iter (Int): number of max iterations (outer)
max_inner (Int): number of iterations for the CG solves
tol (Float64): tolerance for the relative change θ ie norm(θ-θ_)/norm(θ)
damping (Float64): how much to trust iterates (1=full trust)
threshold (Float64): threshold for the residuals

source