API

Standalone

Regression

Standard constructors

MLJLinearModels.LinearRegression — Function

LinearRegression(; fit_intercept)

Objective function: $|Xθ - y|₂²/2$.

source

MLJLinearModels.RidgeRegression — Function

RidgeRegression(; ...)
RidgeRegression(
    λ;
    lambda,
    fit_intercept,
    penalize_intercept,
    scale_penalty_with_samples
)

Objective function: $|Xθ - y|₂²/2 + n⋅λ|θ|₂²/2$, where $n$ is the number of samples size(X, 1). With scale_penalty_with_samples = false the objective function is $|Xθ - y|₂²/2 + λ|θ|₂²/2$.

source

MLJLinearModels.LassoRegression — Function

LassoRegression(; ...)
LassoRegression(
    λ;
    lambda,
    fit_intercept,
    penalize_intercept,
    scale_penalty_with_samples
)

Objective function: $|Xθ - y|₂²/2 + n⋅λ|θ|₁$, where $n$ is the number of samples size(X, 1). With scale_penalty_with_samples = false the objective function is $|Xθ - y|₂²/2 + λ|θ|₁$

source

MLJLinearModels.ElasticNetRegression — Function

ElasticNetRegression(; ...)
ElasticNetRegression(λ; ...)
ElasticNetRegression(
    λ,
    γ;
    lambda,
    gamma,
    fit_intercept,
    penalize_intercept,
    scale_penalty_with_samples
)

Objective function: $|Xθ - y|₂²/2 + n⋅λ|θ|₂²/2 + n⋅γ|θ|₁$, where $n$ is the number of samples size(X, 1). With scale_penalty_with_samples = false the objective function is $|Xθ - y|₂²/2 + λ|θ|₂²/2 + γ|θ|₁$

source

MLJLinearModels.HuberRegression — Function

HuberRegression(; ...)
HuberRegression(δ; ...)
HuberRegression(δ, λ; ...)
HuberRegression(
    δ,
    λ,
    γ;
    delta,
    lambda,
    gamma,
    penalty,
    fit_intercept,
    scale_penalty_with_samples,
    penalize_intercept
)

Huber Regression with objective:

$∑ρ(Xθ - y) + n⋅λ|θ|₂²/2 + n⋅γ|θ|₁$

Where ρ is the Huber function ρ(r) = r²/2if|r|≤δandρ(r)=δ(|r|-δ/2)otherwise andnis the number of samplessize(X, 1). Withscalepenaltywith_samples = falsethe objective function is∑ρ(Xθ - y) + λ|θ|₂²/2 + γ|θ|₁`.

source

MLJLinearModels.QuantileRegression — Function

QuantileRegression(; ...)
QuantileRegression(δ; ...)
QuantileRegression(δ, λ; ...)
QuantileRegression(
    δ,
    λ,
    γ;
    delta,
    lambda,
    gamma,
    penalty,
    fit_intercept,
    scale_penalty_with_samples,
    penalize_intercept
)

Quantile Regression with objective:

$∑ρ(Xθ - y) + n⋅λ|θ|₂²/2 + n⋅γ|θ|₁$

Where ρ is the check function ρ(r) = r(δ - 1(r < 0)) and $n$ is the number of samples size(X, 1). With scale_penalty_with_samples = false the objective function is $∑ρ(Xθ - y) + λ|θ|₂²/2 + γ|θ|₁$.

source

MLJLinearModels.LADRegression — Function

LADRegression(; ...)
LADRegression(λ; ...)
LADRegression(
    λ,
    γ;
    lambda,
    gamma,
    penalty,
    scale_penalty_with_samples,
    fit_intercept,
    penalize_intercept
)

Least Absolute Deviation regression with objective:

$|Xθ - y|₁ + n⋅λ|θ|₂²/2 + n⋅γ|θ|₁$ where $n$ is the number of samples size(X, 1). With scale_penalty_with_samples = false the objective function is $|Xθ - y|₁ + λ|θ|₂²/2 + γ|θ|₁$.

This is a specific type of Quantile Regression with δ=0.5 (median).

source

Generic constructors

MLJLinearModels.GeneralizedLinearRegression — Type

GeneralizedLinearRegression{L<:Loss, P<:Penalty}

Generalized Linear Regression (GLR) model with objective function:

$L(y, Xθ) + P(θ)$

where L is a loss function, P a penalty, y is the vector of observed response, X is the feature matrix and θ the vector of parameters. If scale_penalty_with_samples = true (default) the penalty is automatically scaled with the number of samples.

Special cases include:

OLS regression: L2 loss, no penalty.
Ridge regression: L2 loss, L2 penalty.
Lasso regression: L2 loss, L1 penalty.
Logistic regression: Logit loss, [no,L1,L2] penalty.

source

MLJLinearModels.RobustRegression — Function

RobustRegression(; ...)
RobustRegression(ρ; ...)
RobustRegression(ρ, λ; ...)
RobustRegression(
    ρ,
    λ,
    γ;
    rho,
    lambda,
    gamma,
    penalty,
    fit_intercept,
    scale_penalty_with_samples,
    penalize_intercept
)

Objective function: $∑ρ(Xθ - y) + n⋅λ|θ|₂² + n⋅γ|θ|₁$ where ρ is a given function on the residuals and $n$ is the number of samples size(X, 1). With scale_penalty_with_samples = false the objective function is $∑ρ(Xθ - y) + λ|θ|₂² + γ|θ|₁$.

source

Classification

MLJLinearModels.LogisticRegression — Function

LogisticRegression(; ...)
LogisticRegression(λ; ...)
LogisticRegression(
    λ,
    γ;
    lambda,
    gamma,
    penalty,
    fit_intercept,
    penalize_intercept,
    scale_penalty_with_samples,
    multi_class,
    nclasses
)

Objective function: $L(y, Xθ) + n⋅λ|θ|₂²/2 + n⋅γ|θ|₁$ where L is either the logistic loss in the binary case or the multinomial loss otherwise and $n$ is the number of samples size(X, 1). With scale_penalty_with_samples = false the objective function is $L(y, Xθ) + λ|θ|₂²/2 + γ|θ|₁$.

source

MLJLinearModels.MultinomialRegression — Function

MultinomialRegression(a; kwa...)

Objective function: $L(y, Xθ) + n⋅λ|θ|₂²/2 + n⋅γ|θ|₁$ where L is the multinomial loss and $n$ is the number of samples size(X, 1). With scale_penalty_with_samples = false the objective function is $L(y, Xθ) + λ|θ|₂²/2 + γ|θ|₁$.

source

MLJ Interface

Regressors

MLJLinearModels.LinearRegressor — Type

LinearRegressor

A model type for constructing a linear regressor, based on MLJLinearModels.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

LinearRegressor = @load LinearRegressor pkg=MLJLinearModels

Do model = LinearRegressor() to construct an instance with default hyper-parameters.

This model provides standard linear regression with objective function

$|Xθ - y|₂²/2$

Different solver options exist, as indicated under "Hyperparameters" below.

Training data

In MLJ or MLJBase, bind an instance model to data with

mach = machine(model, X, y)

where:

X is any table of input features (eg, a DataFrame) whose columns have Continuous scitype; check column scitypes with schema(X)
y is the target, which can be any AbstractVector whose element scitype is Continuous; check the scitype with scitype(y)

Train the machine using fit!(mach, rows=...).

Hyperparameters

fit_intercept::Bool: whether to fit the intercept or not. Default: true
solver::Union{Nothing, MLJLinearModels.Solver}: "any instance of MLJLinearModels.Analytical. Use Analytical() for Cholesky and CG()=Analytical(iterative=true) for conjugate-gradient.
If solver = nothing (default) then Analytical() is used. Default: nothing

Example

using MLJ
X, y = make_regression()
mach = fit!(machine(LinearRegressor(), X, y))
predict(mach, X)
fitted_params(mach)

source

MLJLinearModels.RidgeRegressor — Type

RidgeRegressor

A model type for constructing a ridge regressor, based on MLJLinearModels.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

RidgeRegressor = @load RidgeRegressor pkg=MLJLinearModels

Do model = RidgeRegressor() to construct an instance with default hyper-parameters.

Ridge regression is a linear model with objective function

$|Xθ - y|₂²/2 + n⋅λ|θ|₂²/2$

where $n$ is the number of observations.

If scale_penalty_with_samples = false then the objective function is instead

$|Xθ - y|₂²/2 + λ|θ|₂²/2$.

Different solver options exist, as indicated under "Hyperparameters" below.

Training data

In MLJ or MLJBase, bind an instance model to data with

mach = machine(model, X, y)

where:

X is any table of input features (eg, a DataFrame) whose columns have Continuous scitype; check column scitypes with schema(X)
y is the target, which can be any AbstractVector whose element scitype is Continuous; check the scitype with scitype(y)

Train the machine using fit!(mach, rows=...).

Hyperparameters

lambda::Real: strength of the L2 regularization. Default: 1.0
fit_intercept::Bool: whether to fit the intercept or not. Default: true
penalize_intercept::Bool: whether to penalize the intercept. Default: false
scale_penalty_with_samples::Bool: whether to scale the penalty with the number of observations. Default: true
solver::Union{Nothing, MLJLinearModels.Solver}: any instance of MLJLinearModels.Analytical. Use Analytical() for Cholesky and CG()=Analytical(iterative=true) for conjugate-gradient. If solver = nothing (default) then Analytical() is used. Default: nothing

Example

using MLJ
X, y = make_regression()
mach = fit!(machine(RidgeRegressor(), X, y))
predict(mach, X)
fitted_params(mach)

See also LassoRegressor.

source

MLJLinearModels.HuberRegressor — Type

HuberRegressor

A model type for constructing a huber regressor, based on MLJLinearModels.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

HuberRegressor = @load HuberRegressor pkg=MLJLinearModels

Do model = HuberRegressor() to construct an instance with default hyper-parameters.

This model coincides with RobustRegressor, with the exception that the robust loss, rho, is fixed to HuberRho(delta), where delta is a new hyperparameter.

Different solver options exist, as indicated under "Hyperparameters" below.

Training data

In MLJ or MLJBase, bind an instance model to data with

mach = machine(model, X, y)

where:

X is any table of input features (eg, a DataFrame) whose columns have Continuous scitype; check column scitypes with schema(X)
y is the target, which can be any AbstractVector whose element scitype is Continuous; check the scitype with scitype(y)

Train the machine using fit!(mach, rows=...).

Hyperparameters

delta::Real: parameterizes the HuberRho function (radius of the ball within which the loss is a quadratic loss) Default: 0.5
lambda::Real: strength of the regularizer if penalty is :l2 or :l1. Strength of the L2 regularizer if penalty is :en. Default: 1.0
gamma::Real: strength of the L1 regularizer if penalty is :en. Default: 0.0
penalty::Union{String, Symbol}: the penalty to use, either :l2, :l1, :en (elastic net) or :none. Default: :l2
fit_intercept::Bool: whether to fit the intercept or not. Default: true
penalize_intercept::Bool: whether to penalize the intercept. Default: false
scale_penalty_with_samples::Bool: whether to scale the penalty with the number of observations. Default: true
solver::Union{Nothing, MLJLinearModels.Solver}: some instance of MLJLinearModels.S where S is one of: LBFGS, IWLSCG, Newton, NewtonCG, if penalty = :l2, and ProxGrad otherwise.
If solver = nothing (default) then LBFGS() is used, if penalty = :l2, and otherwise ProxGrad(accel=true) (FISTA) is used.
Solver aliases: FISTA(; kwargs...) = ProxGrad(accel=true, kwargs...), ISTA(; kwargs...) = ProxGrad(accel=false, kwargs...) Default: nothing

Example

using MLJ
X, y = make_regression()
mach = fit!(machine(HuberRegressor(), X, y))
predict(mach, X)
fitted_params(mach)

source

MLJLinearModels.QuantileRegressor — Type

QuantileRegressor

A model type for constructing a quantile regressor, based on MLJLinearModels.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

QuantileRegressor = @load QuantileRegressor pkg=MLJLinearModels

Do model = QuantileRegressor() to construct an instance with default hyper-parameters.

This model coincides with RobustRegressor, with the exception that the robust loss, rho, is fixed to QuantileRho(delta), where delta is a new hyperparameter.

Different solver options exist, as indicated under "Hyperparameters" below.

Training data

In MLJ or MLJBase, bind an instance model to data with

mach = machine(model, X, y)

where:

X is any table of input features (eg, a DataFrame) whose columns have Continuous scitype; check column scitypes with schema(X)
y is the target, which can be any AbstractVector whose element scitype is Continuous; check the scitype with scitype(y)

Train the machine using fit!(mach, rows=...).

Hyperparameters

delta::Real: parameterizes the QuantileRho function (indicating the quantile to use with default 0.5 for the median regression) Default: 0.5
lambda::Real: strength of the regularizer if penalty is :l2 or :l1. Strength of the L2 regularizer if penalty is :en. Default: 1.0
gamma::Real: strength of the L1 regularizer if penalty is :en. Default: 0.0
penalty::Union{String, Symbol}: the penalty to use, either :l2, :l1, :en (elastic net) or :none. Default: :l2
fit_intercept::Bool: whether to fit the intercept or not. Default: true
penalize_intercept::Bool: whether to penalize the intercept. Default: false
scale_penalty_with_samples::Bool: whether to scale the penalty with the number of observations. Default: true
solver::Union{Nothing, MLJLinearModels.Solver}: some instance of MLJLinearModels.S where S is one of: LBFGS, IWLSCG, if penalty = :l2, and ProxGrad otherwise.
If solver = nothing (default) then LBFGS() is used, if penalty = :l2, and otherwise ProxGrad(accel=true) (FISTA) is used.
Solver aliases: FISTA(; kwargs...) = ProxGrad(accel=true, kwargs...), ISTA(; kwargs...) = ProxGrad(accel=false, kwargs...) Default: nothing

Example

using MLJ
X, y = make_regression()
mach = fit!(machine(QuantileRegressor(), X, y))
predict(mach, X)
fitted_params(mach)

source

MLJLinearModels.LADRegressor — Type

LADRegressor

A model type for constructing a lad regressor, based on MLJLinearModels.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

LADRegressor = @load LADRegressor pkg=MLJLinearModels

Do model = LADRegressor() to construct an instance with default hyper-parameters.

Least absolute deviation regression is a linear model with objective function

$∑ρ(Xθ - y) + n⋅λ|θ|₂² + n⋅γ|θ|₁$

where $ρ$ is the absolute loss and $n$ is the number of observations.

If scale_penalty_with_samples = false the objective function is instead

$∑ρ(Xθ - y) + λ|θ|₂² + γ|θ|₁$.

Different solver options exist, as indicated under "Hyperparameters" below.

Training data

In MLJ or MLJBase, bind an instance model to data with

mach = machine(model, X, y)

where:

X is any table of input features (eg, a DataFrame) whose columns have Continuous scitype; check column scitypes with schema(X)
y is the target, which can be any AbstractVector whose element scitype is Continuous; check the scitype with scitype(y)

Train the machine using fit!(mach, rows=...).

Hyperparameters

Classifiers

MLJLinearModels.LogisticClassifier — Type

LogisticClassifier

A model type for constructing a logistic classifier, based on MLJLinearModels.jl, and implementing the MLJ model interface.

From MLJ, the type can be imported using

LogisticClassifier = @load LogisticClassifier pkg=MLJLinearModels

Do model = LogisticClassifier() to construct an instance with default hyper-parameters.

This model is more commonly known as "logistic regression". It is a standard classifier for both binary and multiclass classification. The objective function applies either a logistic loss (binary target) or multinomial (softmax) loss, and has a mixed L1/L2 penalty:

$L(y, Xθ) + n⋅λ|θ|₂²/2 + n⋅γ|θ|₁$.

Here $L$ is either MLJLinearModels.LogisticLoss or MLJLinearModels.MultiClassLoss, $λ$ and $γ$ indicate the strength of the L2 (resp. L1) regularization components and $n$ is the number of training observations.

With scale_penalty_with_samples = false the objective function is instead

$L(y, Xθ) + λ|θ|₂²/2 + γ|θ|₁$.

Training data

In MLJ or MLJBase, bind an instance model to data with

mach = machine(model, X, y)

where:

X is any table of input features (eg, a DataFrame) whose columns have Continuous scitype; check column scitypes with schema(X)
y is the target, which can be any AbstractVector whose element scitype is <:OrderedFactor or <:Multiclass; check the scitype with scitype(y)

Train the machine using fit!(mach, rows=...).

Hyperparameters

lambda::Real: strength of the regularizer if penalty is :l2 or :l1 and strength of the L2 regularizer if penalty is :en. Default: eps()
gamma::Real: strength of the L1 regularizer if penalty is :en. Default: 0.0
penalty::Union{String, Symbol}: the penalty to use, either :l2, :l1, :en (elastic net) or :none. Default: :l2
fit_intercept::Bool: whether to fit the intercept or not. Default: true
penalize_intercept::Bool: whether to penalize the intercept. Default: false
scale_penalty_with_samples::Bool: whether to scale the penalty with the number of samples. Default: true
solver::Union{Nothing, MLJLinearModels.Solver}: some instance of MLJLinearModels.S where S is one of: LBFGS, Newton, NewtonCG, ProxGrad; but subject to the following restrictions:
- If penalty = :l2, ProxGrad is disallowed. Otherwise, ProxGrad is the only option.
- Unless scitype(y) <: Finite{2} (binary target) Newton is disallowed.
If solver = nothing (default) then ProxGrad(accel=true) (FISTA) is used, unless gamma = 0, in which case LBFGS() is used.
Solver aliases: FISTA(; kwargs...) = ProxGrad(accel=true, kwargs...), ISTA(; kwargs...) = ProxGrad(accel=false, kwargs...) Default: nothing

Example

using MLJ
X, y = make_blobs(centers = 2)
mach = fit!(machine(LogisticClassifier(), X, y))
predict(mach, X)
fitted_params(mach)