Utilities

MLJBase.fit_only!(mach::Machine; rows=nothing, verbosity=1, force=false)

Without mutating any other machine on which it may depend, perform one of the following actions to the machine mach, using the data and model bound to it, and restricting the data to rows if specified:

• Ab initio training. Ignoring any previous learned parameters and cache, compute and store new learned parameters. Increment mach.state.

• Training update. Making use of previous learned parameters and/or cache, replace or mutate existing learned parameters. The effect is the same (or nearly the same) as in ab initio training, but may be faster or use less memory, assuming the model supports an update option (implements MLJBase.update). Increment mach.state.

• No-operation. Leave existing learned parameters untouched. Do not increment mach.state.

Training action logic

For the action to be a no-operation, either mach.frozen == true or or none of the following apply:

• (i) mach has never been trained (mach.state == 0).

• (ii) force == true.

• (iii) The state of some other machine on which mach depends has changed since the last time mach was trained (ie, the last time mach.state was last incremented).

• (iv) The specified rows have changed since the last retraining and mach.model does not have Static type.

• (v) mach.model has changed since the last retraining.

In any of the cases (i) - (iv), mach is trained ab initio. If only (v) fails, then a training update is applied.

To freeze or unfreeze mach, use freeze!(mach) or thaw!(mach).

Implementation detail

The data to which a machine is bound is stored in mach.args. Each element of args is either a Node object, or, in the case that concrete data was bound to the machine, it is concrete data wrapped in a Source node. In all cases, to obtain concrete data for actual training, each argument N is called, as in N() or N(rows=rows), and either MLJBase.fit (ab initio training) or MLJBase.update (training update) is dispatched on mach.model and this data. See the "Adding models for general use" section of the MLJ documentation for more on these lower-level training methods.

freeze!(mach)

Freeze the machine mach so that it will never be retrained (unless thawed).

See also thaw!.

N = glb(mach::Machine{<:Surrogate})

A greatest lower bound for the nodes appearing in the signature of mach.

Private method.

machine(model, args...; cache=true)

Construct a Machine object binding a model, storing hyper-parameters of some machine learning algorithm, to some data, args. When building a learning network, Node objects can be substituted for concrete data. Specify cache=false to prioritize memory managment over speed, and to guarantee data anonymity when serializing composite models.

machine(Xs; oper1=node1, oper2=node2)
machine(Xs, ys; oper1=node1, oper2=node2)
machine(Xs, ys, extras...; oper1=node1, oper2=node2, ...)

Construct a special machine called a learning network machine, that "wraps" a learning network, usually in preparation to export the network as a stand-alone composite model type. The keyword arguments declare what nodes are called when operations, such as predict and transform, are called on the machine.

In addition to the operations named in the constructor, the methods fit!, report, and fitted_params can be applied as usual to the machine constructed.

machine(Probablistic(), args...; kwargs...)
machine(Deterministic(), args...; kwargs...)
machine(Unsupervised(), args...; kwargs...)
machine(Static(), args...; kwargs...)

Same as above, but specifying explicitly the kind of model the learning network is to meant to represent.

Learning network machines are not to be confused with an ordinary machine that happens to be bound to a stand-alone composite model (i.e., an exported learning network).

Examples

Supposing a supervised learning network's final predictions are obtained by calling a node yhat, then the code

mach = machine(Deterministic(), Xs, ys; predict=yhat)
fit!(mach; rows=train)
predictions = predict(mach, Xnew) # `Xnew` concrete data

is equivalent to

fit!(yhat, rows=train)
predictions = yhat(Xnew)

Here Xs and ys are the source nodes receiving, respectively, the input and target data.

In a unsupervised learning network for clustering, with single source node Xs for inputs, and in which the node Xout delivers the output of dimension reduction, and yhat the class labels, one can write

mach = machine(Unsupervised(), Xs; transform=Xout, predict=yhat)
fit!(mach)
transformed = transform(mach, Xnew) # `Xnew` concrete data
predictions = predict(mach, Xnew)

which is equivalent to

fit!(Xout)
fit!(yhat)
transformed = Xout(Xnew)
predictions = yhat(Xnew)

report(mach)

Return the report for a machine mach that has been fit!, for example the coefficients in a linear model.

This is a named tuple and human-readable if possible.

If mach is a machine for a composite model, such as a model constructed using @pipeline, then the returned named tuple has the composite type's field names as keys. The corresponding value is the report for the machine in the underlying learning network bound to that model. (If multiple machines share the same model, then the value is a vector.)

using MLJ
@load LinearBinaryClassifier pkg=GLM
X, y = @load_crabs;
pipe = @pipeline Standardizer LinearBinaryClassifier
mach = machine(pipe, X, y) |> fit!

julia> report(mach).linear_binary_classifier
(deviance = 3.8893386087844543e-7,
 dof_residual = 195.0,
 stderror = [18954.83496713119, 6502.845740757159, 48484.240246060406, 34971.131004997274, 20654.82322484894, 2111.1294584763386],
 vcov = [3.592857686311793e8 9.122732393971942e6 … -8.454645589364915e7 5.38856837634321e6; 9.122732393971942e6 4.228700272808351e7 … -4.978433790526467e7 -8.442545425533723e6; … ; -8.454645589364915e7 -4.978433790526467e7 … 4.2662172244975924e8 2.1799125705781363e7; 5.38856837634321e6 -8.442545425533723e6 … 2.1799125705781363e7 4.456867590446599e6],)

Additional keys, machines and report_given_machine, give a list of all machines in the underlying network, and a dictionary of reports keyed on those machines.

```

return!(mach::Machine{<:Surrogate}, model, verbosity)

The last call in custom code defining the MLJBase.fit method for a new composite model type. Here model is the instance of the new type appearing in the MLJBase.fit signature, while mach is a learning network machine constructed using model. Not relevant when defining composite models using @pipeline or @from_network.

For usage, see the example given below. Specificlly, the call does the following:

• Determines which fields of model point to model instances in the learning network wrapped by mach, for recording in an object called cache, for passing onto the MLJ logic that handles smart updating (namely, an MLJBase.update fallback for composite models).

• Calls fit!(mach, verbosity=verbosity).

• Moves any data in source nodes of the learning network into cache (for data-anonymization purposes).

• Records a copy of model in cache.

• Returns cache and outcomes of training in an appropriate form (specifically, (mach.fitresult, cache, mach.report); see Adding Models for General Use for technical details.)

Example

The following code defines, "by hand", a new model type MyComposite for composing standardization (whitening) with a deterministic regressor:

mutable struct MyComposite <: DeterministicComposite
    regressor
end

function MLJBase.fit(model::MyComposite, verbosity, X, y)
    Xs = source(X)
    ys = source(y)

    mach1 = machine(Standardizer(), Xs)
    Xwhite = transform(mach1, Xs)

    mach2 = machine(model.regressor, Xwhite, ys)
    yhat = predict(mach2, Xwhite)

    mach = machine(Deterministic(), Xs, ys; predict=yhat)
    return!(mach, model, verbosity)
end

thaw!(mach)

Unfreeze the machine mach so that it can be retrained.

See also freeze!.

fitted_params(mach)

Return the learned parameters for a machine mach that has been fit!, for example the coefficients in a linear model.

This is a named tuple and human-readable if possible.

If mach is a machine for a composite model, such as a model constructed using @pipeline, then the returned named tuple has the composite type's field names as keys. The corresponding value is the fitted parameters for the machine in the underlying learning network bound to that model. (If multiple machines share the same model, then the value is a vector.)

using MLJ
@load LogisticClassifier pkg=MLJLinearModels
X, y = @load_crabs;
pipe = @pipeline Standardizer LogisticClassifier
mach = machine(pipe, X, y) |> fit!

julia> fitted_params(mach).logistic_classifier
(classes = CategoricalArrays.CategoricalValue{String,UInt32}["B", "O"],
 coefs = Pair{Symbol,Float64}[:FL => 3.7095037897680405, :RW => 0.1135739140854546, :CL => -1.6036892745322038, :CW => -4.415667573486482, :BD => 3.238476051092471],
 intercept = 0.0883301599726305,)

Additional keys, machines and fitted_params_given_machine, give a list of all machines in the underlying network, and a dictionary of fitted parameters keyed on those machines.

```

fit!(mach::Machine{<:Surrogate};
     rows=nothing,
     acceleration=CPU1(),
     verbosity=1,
     force=false))

Train the complete learning network wrapped by the machine mach.

More precisely, if s is the learning network signature used to construct mach, then call fit!(N), where N = glb(values(s)...) is a greatest lower bound on the nodes appearing in the signature. For example, if s = (predict=yhat, transform=W), then call fit!(glb(yhat, W)). Here glb is tuple overloaded for nodes.

See also machine

fit!(mach::Machine, rows=nothing, verbosity=1, force=false)

Fit the machine mach. In the case that mach has Node arguments, first train all other machines on which mach depends.

To attempt to fit a machine without touching any other machine, use fit_only!. For more on the internal logic of fitting see fit_only!

replace(mach, a1=>b1, a2=>b2, ...; empty_unspecified_sources=false)

Create a deep copy of a learning network machine mach but replacing any specified sources and models a1, a2, ... of the original underlying network with b1, b2, ....

If empty_unspecified_sources=true then any source nodes not specified are replaced with empty source nodes.

ancestors(mach::Machine; self=false)

All ancestors of mach, including mach if self=true.

model_supertype(signature)

Return, if this can be deduced, which of Deterministic, Probabilistic and Unsupervised is the appropriate supertype for a composite model obtained by exporting a learning network with the specified signature.

A learning network signature is a named tuple, such as (predict=yhat, transfrom=W), specifying what nodes of the network are called to produce output of each operation represented by the keys, in an exported version of the network.

If a supertype cannot be deduced, nothing is returned.

If the network with given signature is not exportable, this method will not error but it will not a give meaningful return value either.

Private method.

MLJ.save(filename, mach::Machine; kwargs...)
MLJ.save(io, mach::Machine; kwargs...)

MLJBase.save(filename, mach::Machine; kwargs...)
MLJBase.save(io, mach::Machine; kwargs...)

Serialize the machine mach to a file with path filename, or to an input/output stream io (at least IOBuffer instances are supported).

The format is JLSO (a wrapper for julia native or BSON serialization). For some model types, a custom serialization will be additionally performed.

Keyword arguments

These keyword arguments are passed to the JLSO serializer:

See https://github.com/invenia/JLSO.jl for details.

Any additional keyword arguments are passed to model-specific serializers.

Machines are de-serialized using the machine constructor as shown in the example below. Data (or nodes) may be optionally passed to the constructor for retraining on new data using the saved model.

Example

using MLJ
tree = @load DecisionTreeClassifier
X, y = @load_iris
mach = fit!(machine(tree, X, y))

MLJ.save("tree.jlso", mach, compression=:none)
mach_predict_only = machine("tree.jlso")
predict(mach_predict_only, X)

mach2 = machine("tree.jlso", selectrows(X, 1:100), y[1:100])
predict(mach2, X) # same as above

fit!(mach2) # saved learned parameters are over-written
predict(mach2, X) # not same as above

# using a buffer:
io = IOBuffer()
MLJ.save(io, mach)
seekstart(io)
predict_only_mach = machine(io)
predict(predict_only_mach, X)

color_off()

Suppress color and bold output at the REPL for displaying MLJ objects.

color_on()

Enable color and bold output at the REPL, for enhanced display of MLJ objects.

@constant x = value

Equivalent to const x = value but registers the binding thus:

MLJBase.HANDLE_GIVEN_ID[objectid(value)] = :x

Registered objects get displayed using the variable name to which it was bound in calls to show(x), etc.

WARNING: As with any const declaration, binding x to new value of the same type is not prevented and the registration will not be updated.

@more

Entered at the REPL, equivalent to show(ans, 100). Use to get a recursive description of all fields of the last REPL value.

_recursive_show(stream, object, current_depth, depth)

Generate a table of the field values of the MLJType object, dislaying each value by calling the method _show on it. The behaviour of _show(stream, f) is as follows:

1. If f is itself a MLJType object, then its short form is shown

and _recursive_show generates as separate table for each of its field values (and so on, up to a depth of argument depth).

1. Otherwise f is displayed as "(omitted T)" where T = typeof(f),

unless istoobig(f) is false (the istoobig fall-back for arbitrary types being true). In the latter case, the long (ie, MIME"plain/text") form of f is shown. To override this behaviour, overload the _show method for the type in question.

to display abbreviated versions of integers

return abbreviated object id (as string) or it's registered handle (as string) if this exists

LossFunctions.DWDMarginLoss

A measure type for distance weighted discrimination loss, which includes the instance(s), dwd_margin_loss.

DWDMarginLoss()(ŷ, y)
DWDMarginLoss()(ŷ, y, w)

Evaluate the default instance of DWDMarginLoss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

Constructor signature: DWDMarginLoss(; q=1.0)

For more information, run info(DWDMarginLoss).

LossFunctions.ExpLoss

A measure type for exp loss, which includes the instance(s), exp_loss.

ExpLoss()(ŷ, y)
ExpLoss()(ŷ, y, w)

Evaluate the exp loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(ExpLoss).

LossFunctions.HuberLoss

A measure type for huber loss, which includes the instance(s), huber_loss.

HuberLoss()(ŷ, y)
HuberLoss()(ŷ, y, w)

Evaluate the default instance of HuberLoss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(HuberLoss).

LossFunctions.L1EpsilonInsLoss

A measure type for l1 ϵ-insensitive loss, which includes the instance(s), l1_epsilon_ins_loss.

L1EpsilonInsLoss()(ŷ, y)
L1EpsilonInsLoss()(ŷ, y, w)

Evaluate the default instance of L1EpsilonInsLoss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

Constructor signature: L1EpsilonInsLoss(; ϵ=1.0)

For more information, run info(L1EpsilonInsLoss).

LossFunctions.L1HingeLoss

A measure type for l1 hinge loss, which includes the instance(s), l1_hinge_loss.

L1HingeLoss()(ŷ, y)
L1HingeLoss()(ŷ, y, w)

Evaluate the l1 hinge loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(L1HingeLoss).

LossFunctions.L2EpsilonInsLoss

A measure type for l2 ϵ-insensitive loss, which includes the instance(s), l2_epsilon_ins_loss.

L2EpsilonInsLoss()(ŷ, y)
L2EpsilonInsLoss()(ŷ, y, w)

Evaluate the default instance of L2EpsilonInsLoss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

Constructor signature: L2EpsilonInsLoss(; ϵ=1.0)

For more information, run info(L2EpsilonInsLoss).

LossFunctions.L2HingeLoss

A measure type for l2 hinge loss, which includes the instance(s), l2_hinge_loss.

L2HingeLoss()(ŷ, y)
L2HingeLoss()(ŷ, y, w)

Evaluate the l2 hinge loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(L2HingeLoss).

LossFunctions.L2MarginLoss

A measure type for l2 margin loss, which includes the instance(s), l2_margin_loss.

L2MarginLoss()(ŷ, y)
L2MarginLoss()(ŷ, y, w)

Evaluate the l2 margin loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(L2MarginLoss).

LossFunctions.LPDistLoss

A measure type for lp dist loss, which includes the instance(s), lp_dist_loss.

LPDistLoss()(ŷ, y)
LPDistLoss()(ŷ, y, w)

Evaluate the lp dist loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

Constructor signature: LPDistLoss(; P=2)

For more information, run info(LPDistLoss).

LossFunctions.LogitDistLoss

A measure type for logit dist loss, which includes the instance(s), logit_dist_loss.

LogitDistLoss()(ŷ, y)
LogitDistLoss()(ŷ, y, w)

Evaluate the logit dist loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(LogitDistLoss).

LossFunctions.LogitMarginLoss

A measure type for logit margin loss, which includes the instance(s), logit_margin_loss.

LogitMarginLoss()(ŷ, y)
LogitMarginLoss()(ŷ, y, w)

Evaluate the logit margin loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(LogitMarginLoss).

LossFunctions.ModifiedHuberLoss

A measure type for modified huber loss, which includes the instance(s), modified_huber_loss.

ModifiedHuberLoss()(ŷ, y)
ModifiedHuberLoss()(ŷ, y, w)

Evaluate the modified huber loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(ModifiedHuberLoss).

LossFunctions.PerceptronLoss

A measure type for perceptron loss, which includes the instance(s), perceptron_loss.

PerceptronLoss()(ŷ, y)
PerceptronLoss()(ŷ, y, w)

Evaluate the perceptron loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(PerceptronLoss).

LossFunctions.PeriodicLoss

A measure type for periodic loss, which includes the instance(s), periodic_loss.

PeriodicLoss()(ŷ, y)
PeriodicLoss()(ŷ, y, w)

Evaluate the default instance of PeriodicLoss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(PeriodicLoss).

LossFunctions.QuantileLoss

A measure type for quantile loss, which includes the instance(s), quantile_loss.

QuantileLoss()(ŷ, y)
QuantileLoss()(ŷ, y, w)

Evaluate the default instance of QuantileLoss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

Constructor signature: QuantileLoss(; τ=0.7)

For more information, run info(QuantileLoss).

LossFunctions.SigmoidLoss

A measure type for sigmoid loss, which includes the instance(s), sigmoid_loss.

SigmoidLoss()(ŷ, y)
SigmoidLoss()(ŷ, y, w)

Evaluate the sigmoid loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(SigmoidLoss).

LossFunctions.SmoothedL1HingeLoss

A measure type for smoothed l1 hinge loss, which includes the instance(s), smoothed_l1_hinge_loss.

SmoothedL1HingeLoss()(ŷ, y)
SmoothedL1HingeLoss()(ŷ, y, w)

Evaluate the default instance of SmoothedL1HingeLoss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

Constructor signature: SmoothedL1HingeLoss(; γ=1.0)

For more information, run info(SmoothedL1HingeLoss).

LossFunctions.ZeroOneLoss

A measure type for zero one loss, which includes the instance(s), zero_one_loss.

ZeroOneLoss()(ŷ, y)
ZeroOneLoss()(ŷ, y, w)

Evaluate the zero one loss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

See above for original LossFunctions.jl documentation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a probabilistic prediction.

For more information, run info(ZeroOneLoss).

MLJBase.Accuracy

A measure type for accuracy, which includes the instance(s), accuracy.

Accuracy()(ŷ, y)
Accuracy()(ŷ, y, w)

Evaluate the accuracy on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

Accuracy is proportion of correct predictions ŷ[i] that match the ground truth y[i] observations. This metric is invariant to class reordering.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite} (multiclass classification); ŷ must be a deterministic prediction.

For more information, run info(Accuracy).

MLJBase.AreaUnderCurve

A measure type for area under the ROC, which includes the instance(s), area_under_curve, auc.

AreaUnderCurve()(ŷ, y)

Evaluate the area under the ROC on observations ŷ, given ground truth values y.

Returns the area under the ROC (receiver operator characteristic) This metric is invariant to class reordering.

Requires scitype(y) to be a subtype of AbstractArray{var"#s584",1} where var"#s584"<:ScientificTypes.Finite{2}; ŷ must be a probabilistic prediction.

For more information, run info(AreaUnderCurve).

MLJBase.BalancedAccuracy

A measure type for balanced accuracy, which includes the instance(s), balanced_accuracy, bacc, bac.

BalancedAccuracy()(ŷ, y)
BalancedAccuracy()(ŷ, y, w)

Evaluate the balanced accuracy on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

Balanced accuracy compensates standard Accuracy for class imbalance. See https://en.wikipedia.org/wiki/Precisionandrecall#Imbalanced_data. This metric is invariant to class reordering.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite} (multiclass classification); ŷ must be a deterministic prediction.

For more information, run info(BalancedAccuracy).

MLJBase.BrierLoss

A measure type for Brier loss (a.k.a. quadratic loss), which includes the instance(s), brier_loss.

BrierLoss()(ŷ, y)
BrierLoss()(ŷ, y, w)

Evaluate the Brier loss (a.k.a. quadratic loss) on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

If p(y) is the predicted probability for a single observation y, and C all possible classes, then the corresponding Brier score for that observation is given by

$\left(\sum_{η ∈ C} p(η)^2\right) - 2p(y) + 1$

Warning. In Brier's original 1950 paper, what is implemented here is called a "loss". It is, however, a "score" in the contemporary use of that term: smaller is better (with 0 optimal, and all other values positive). Note also the present implementation does not treat the binary case as special, so that the loss may differ, in that case, by a factor of two from usage elsewhere.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite} (multiclass classification); ŷ must be a probabilistic prediction.

For more information, run info(BrierLoss).

MLJBase.BrierScore

A measure type for Brier score (a.k.a. quadratic score), which includes the instance(s), brier_score.

BrierScore()(ŷ, y)
BrierScore()(ŷ, y, w)

Evaluate the Brier score (a.k.a. quadratic score) on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

If p(y) is the predicted probability for a single observation y, and C all possible classes, then the corresponding Brier score for that observation is given by

$2p(y) - \left(\sum_{η ∈ C} p(η)^2\right) - 1$

Warning. BrierScore() is a "score" in the sense that bigger is better (with 0 optimal, and all other values negative). In Brier's original 1950 paper, and many other places, it has the opposite sign, despite the name. Moreover, the present implementation does not treat the binary case as special, so that the score may differ, in that case, by a factor of two from usage elsewhere.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite} (multiclass classification); ŷ must be a probabilistic prediction.

For more information, run info(BrierScore).

MLJBase.ConfusionMatrix

A measure type for confusion matrix, which includes the instance(s), confusion_matrix, confmat.

ConfusionMatrix()(ŷ, y)

Evaluate the default instance of ConfusionMatrix on observations ŷ, given ground truth values y.

If r is the return value, then the raw confusion matrix is r.mat, whose rows correspond to predictions, and columns to ground truth. The ordering follows that of levels(y).

Use ConfusionMatrix(perm=[2, 1]) to reverse the class order for binary data. For more than two classes, specify an appropriate permutation, as in ConfusionMatrix(perm=[2, 3, 1]).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(ConfusionMatrix).

MLJBase.FScore

A measure type for F-Score, which includes the instance(s), f1score.

FScore()(ŷ, y)

Evaluate the default instance of FScore on observations ŷ, given ground truth values y.

This is the one-parameter generalization, $F_β$, of the F-measure or balanced F-score.

https://en.wikipedia.org/wiki/F1_score

Constructor signature: FScore(; β=1.0, rev=true).

By default, the second element of levels(y) is designated as true. To reverse roles, specify rev=true.

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

Constructor signature: FScore(β=1.0, rev=false).

For more information, run info(FScore).

MLJBase.FalseDiscoveryRate

A measure type for false discovery rate, which includes the instance(s), false_discovery_rate, falsediscovery_rate, fdr.

FalseDiscoveryRate()(ŷ, y)

Evaluate the default instance of FalseDiscoveryRate on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use FalseDiscoveryRate(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(FalseDiscoveryRate).

MLJBase.FalseNegative

A measure type for number of false negatives, which includes the instance(s), false_negative, falsenegative.

FalseNegative()(ŷ, y)

Evaluate the default instance of FalseNegative on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use FalseNegative(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(FalseNegative).

MLJBase.FalseNegativeRate

A measure type for false negative rate, which includes the instance(s), false_negative_rate, falsenegative_rate, fnr, miss_rate.

FalseNegativeRate()(ŷ, y)

Evaluate the default instance of FalseNegativeRate on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use FalseNegativeRate(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(FalseNegativeRate).

MLJBase.FalsePositive

A measure type for number of false positives, which includes the instance(s), false_positive, falsepositive.

FalsePositive()(ŷ, y)

Evaluate the default instance of FalsePositive on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use FalsePositive(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(FalsePositive).

MLJBase.FalsePositiveRate

A measure type for false positive rate, which includes the instance(s), false_positive_rate, falsepositive_rate, fpr, fallout.

FalsePositiveRate()(ŷ, y)

Evaluate the default instance of FalsePositiveRate on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use FalsePositiveRate(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(FalsePositiveRate).

MLJBase.LPLoss

A measure type for lp loss, which includes the instance(s), l1, l2.

LPLoss()(ŷ, y)
LPLoss()(ŷ, y, w)

Evaluate the default instance of LPLoss on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

Constructor signature: LPLoss(p=2). Reports |ŷ[i] - y[i]|^p for every index i.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(LPLoss).

MLJBase.LogCoshLoss

A measure type for log cosh loss, which includes the instance(s), log_cosh, log_cosh_loss.

LogCoshLoss()(ŷ, y)

Evaluate the log cosh loss on observations ŷ, given ground truth values y.

Reports $\log(\cosh(ŷᵢ-yᵢ))$ for each index i.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(LogCoshLoss).

MLJBase.LogLoss

A measure type for log loss, which includes the instance(s), log_loss, cross_entropy.

LogLoss()(ŷ, y)

Evaluate the default instance of LogLoss on observations ŷ, given ground truth values y.

Since the score is undefined in the case that the true observation is predicted to occur with probability zero, probablities are clipped between tol and 1-tol, where tol is a constructor key-word argument.

If sᵢ is the predicted probability for the true class yᵢ then the score for that example is given by

-log(clamp(sᵢ, tol), 1 - tol)

A score is reported for every observation.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite} (multiclass classification); ŷ must be a probabilistic prediction.

For more information, run info(LogLoss).

MLJBase.MatthewsCorrelation

A measure type for matthews correlation, which includes the instance(s), matthews_correlation, mcc.

MatthewsCorrelation()(ŷ, y)

Evaluate the matthews correlation on observations ŷ, given ground truth values y.

https://en.wikipedia.org/wiki/Matthewscorrelationcoefficient This metric is invariant to class reordering.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite{2}} (binary classification); ŷ must be a deterministic prediction.

For more information, run info(MatthewsCorrelation).

MLJBase.MeanAbsoluteError

A measure type for mean absolute error, which includes the instance(s), mae, mav, mean_absolute_error, mean_absolute_value.

MeanAbsoluteError()(ŷ, y)
MeanAbsoluteError()(ŷ, y, w)

Evaluate the mean absolute error on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

$\text{mean absolute error} = n^{-1}∑ᵢ|yᵢ-ŷᵢ|$ or $\text{mean absolute error} = n^{-1}∑ᵢwᵢ|yᵢ-ŷᵢ|$

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(MeanAbsoluteError).

MLJBase.MeanAbsoluteProportionalError

A measure type for mean absolute proportional error, which includes the instance(s), mape.

MeanAbsoluteProportionalError()(ŷ, y)

Evaluate the default instance of MeanAbsoluteProportionalError on observations ŷ, given ground truth values y.

Constructor key-word arguments: tol (default = eps()).

$\text{mean absolute proportional error} = m^{-1}∑ᵢ|{(yᵢ-ŷᵢ) \over yᵢ}|$

where the sum is over indices such that abs(yᵢ) > tol and m is the number of such indices.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(MeanAbsoluteProportionalError).

MLJBase.MisclassificationRate

A measure type for misclassification rate, which includes the instance(s), misclassification_rate, mcr.

MisclassificationRate()(ŷ, y)
MisclassificationRate()(ŷ, y, w)

Evaluate the misclassification rate on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

A confusion matrix can also be passed as argument. This metric is invariant to class reordering.

Requires scitype(y) to be a subtype of AbstractArray{<:Finite} (multiclass classification); ŷ must be a deterministic prediction.

For more information, run info(MisclassificationRate).

MLJBase.NegativePredictiveValue

A measure type for negative predictive value, which includes the instance(s), negative_predictive_value, negativepredictive_value, npv.

NegativePredictiveValue()(ŷ, y)

Evaluate the default instance of NegativePredictiveValue on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use NegativePredictiveValue(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(NegativePredictiveValue).

MLJBase.Precision

A measure type for precision (a.k.a. positive predictive value), which includes the instance(s), positive_predictive_value, ppv, positivepredictive_value, precision.

Precision()(ŷ, y)

Evaluate the default instance of Precision on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use Precision(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(Precision).

MLJBase.RootMeanSquaredError

A measure type for root mean squared error, which includes the instance(s), rms, rmse, root_mean_squared_error.

RootMeanSquaredError()(ŷ, y)
RootMeanSquaredError()(ŷ, y, w)

Evaluate the root mean squared error on observations ŷ, given ground truth values y. Optionally specify per-sample weights, w.

$\text{root mean squared error} = \sqrt{n^{-1}∑ᵢ|yᵢ-ŷᵢ|^2}$ or $\text{root mean squared error} = \sqrt{\frac{∑ᵢwᵢ|yᵢ-ŷᵢ|^2}{∑ᵢwᵢ}}$

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(RootMeanSquaredError).

MLJBase.RootMeanSquaredLogError

A measure type for root mean squared log error, which includes the instance(s), rmsl, rmsle, root_mean_squared_log_error.

RootMeanSquaredLogError()(ŷ, y)

Evaluate the root mean squared log error on observations ŷ, given ground truth values y.

$\text{root mean squared log error} = n^{-1}∑ᵢ\log\left({yᵢ \over ŷᵢ}\right)$

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

See also rmslp1.

For more information, run info(RootMeanSquaredLogError).

MLJBase.RootMeanSquaredLogProportionalError

A measure type for root mean squared log proportional error, which includes the instance(s), rmslp1.

RootMeanSquaredLogProportionalError()(ŷ, y)

Evaluate the default instance of RootMeanSquaredLogProportionalError on observations ŷ, given ground truth values y.

Constructor signature: RootMeanSquaredLogProportionalError(; offset = 1.0).

$\text{root mean squared log proportional error} = n^{-1}∑ᵢ\log\left({yᵢ + \text{offset} \over ŷᵢ + \text{offset}}\right)$

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

See also rmsl.

For more information, run info(RootMeanSquaredLogProportionalError).

MLJBase.RootMeanSquaredProportionalError

A measure type for root mean squared proportional error, which includes the instance(s), rmsp.

RootMeanSquaredProportionalError()(ŷ, y)

Evaluate the default instance of RootMeanSquaredProportionalError on observations ŷ, given ground truth values y.

Constructor keyword arguments: tol (default = eps()).

$\text{root mean squared proportional error} = m^{-1}∑ᵢ \left({yᵢ-ŷᵢ \over yᵢ}\right)^2$

where the sum is over indices such that abs(yᵢ) > tol and m is the number of such indices.

Requires scitype(y) to be a subtype of Union{AbstractArray{ScientificTypes.Continuous,1}, AbstractArray{ScientificTypes.Count,1}}; ŷ must be a deterministic prediction.

For more information, run info(RootMeanSquaredProportionalError).

MLJBase.TrueNegative

A measure type for number of true negatives, which includes the instance(s), true_negative, truenegative.

TrueNegative()(ŷ, y)

Evaluate the default instance of TrueNegative on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use TrueNegative(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(TrueNegative).

MLJBase.TrueNegativeRate

A measure type for true negative rate, which includes the instance(s), true_negative_rate, truenegative_rate, tnr, specificity, selectivity.

TrueNegativeRate()(ŷ, y)

Evaluate the default instance of TrueNegativeRate on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use TrueNegativeRate(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(TrueNegativeRate).

MLJBase.TruePositive

A measure type for number of true positives, which includes the instance(s), true_positive, truepositive.

TruePositive()(ŷ, y)

Evaluate the default instance of TruePositive on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use TruePositive(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(TruePositive).

MLJBase.TruePositiveRate

A measure type for true positive rate (a.k.a recall), which includes the instance(s), true_positive_rate, truepositive_rate, tpr, sensitivity, recall, hit_rate.

TruePositiveRate()(ŷ, y)

Evaluate the default instance of TruePositiveRate on observations ŷ, given ground truth values y.

Assigns false to first element of levels(y). To reverse roles, use TruePositiveRate(rev=true).

Requires scitype(y) to be a subtype of AbstractArray{<:OrderedFactor{2}} (binary classification where choice of "true" effects the measure); ŷ must be a deterministic prediction.

For more information, run info(TruePositiveRate).

flat_values(t::NamedTuple)

View a nested named tuple t as a tree and return, as a tuple, the values at the leaves, in the order they appear in the original tuple.

julia> t = (X = (x = 1, y = 2), Y = 3)
julia> flat_values(t)
(1, 2, 3)

metadata_measure(T; kw...)

Helper function to write the metadata for a single measure.

recursive_getproperty(object, nested_name::Expr)

Call getproperty recursively on object to extract the value of some nested property, as in the following example:

julia> object = (X = (x = 1, y = 2), Y = 3)
julia> recursive_getproperty(object, :(X.y))
2

recursively_setproperty!(object, nested_name::Expr, value)

Set a nested property of an object to value, as in the following example:

julia> mutable struct Foo
           X
           Y
       end

julia> mutable struct Bar
           x
           y
       end

julia> object = Foo(Bar(1, 2), 3)
Foo(Bar(1, 2), 3)

julia> recursively_setproperty!(object, :(X.y), 42)
42

julia> object
Foo(Bar(1, 42), 3)

unwind(iterators...)

Represent all possible combinations of values generated by iterators as rows of a matrix A. In more detail, A has one column for each iterator in iterators and one row for each distinct possible combination of values taken on by the iterators. Elements in the first column cycle fastest, those in the last clolumn slowest.

Example

julia> iterators = ([1, 2], ["a","b"], ["x", "y", "z"]);
julia> MLJTuning.unwind(iterators...)
12×3 Array{Any,2}:
 1  "a"  "x"
 2  "a"  "x"
 1  "b"  "x"
 2  "b"  "x"
 1  "a"  "y"
 2  "a"  "y"
 1  "b"  "y"
 2  "b"  "y"
 1  "a"  "z"
 2  "a"  "z"
 1  "b"  "z"
 2  "b"  "z"

permuterows(obj, perm)

Internal function to return a vector or matrix with permuted rows given the permutation perm.

available_name(modl::Module, name::Symbol)

Function to replace, if necessary, a given name with a modified one that ensures it is not the name of any existing object in the global scope of modl. Modifications are created with numerical suffixes.

check_dimension(X, Y)

Check that two vectors or matrices have matching dimensions

chunks(range, n)

Split an AbstractRange into n subranges of approximately equal length.

Example

julia> collect(chunks(1:5, 2))
2-element Array{UnitRange{Int64},1}:
 1:3
 4:5

**Private method**

shuffle_rows(X, Y, ...; rng=)

Return a shuffled view of a vector or matrix X (or set of such) using a random permutation (which can be seeded specifying rng).