Measures

In MLJ loss functions, scoring rules, sensitivities, and so on, are collectively referred to as measures. Presently, MLJ includes a few built-in measures, provides support for the loss functions in the LossFunctions library, and allows for users to define their own custom measures.

Providing further measures for probabilistic predictors, such as proper scoring rules, is a work in progress.

Built-in measures

These measures all have the common calling syntax measure(ŷ, y) or measure(ŷ, y, w), where y iterates over observations of some target variable, and ŷ iterates over predictions (Distribution or Sampler objects in the probabilistic case). Here w is an optional vector of sample weights, which can be provided when the measure supports this.

julia> using MLJ

julia> y = [1, 2, 3, 4]
4-element Array{Int64,1}:
 1
 2
 3
 4

julia> ŷ = [2, 3, 3, 3]
4-element Array{Int64,1}:
 2
 3
 3
 3

julia> w = [1, 2, 2, 1]
4-element Array{Int64,1}:
 1
 2
 2
 1

julia> rms(ŷ, y) # reports an aggregrate loss
0.8660254037844386

julia> l1(ŷ, y, w) # reports per observation losses
4-element Array{Float64,1}:
 0.6666666666666666
 1.3333333333333333
 0.0
 0.6666666666666666

julia> y = categorical(["male", "female", "female"])
3-element CategoricalArrays.CategoricalArray{String,1,UInt32}:
 "male"
 "female"
 "female"

julia> male = y[1]; female = y[2];

julia> d = UnivariateFinite([male, female], [0.55, 0.45]);

julia> ŷ = [d, d, d];

julia> cross_entropy(ŷ, y)
3-element Array{Float64,1}:
 0.5978370007556204
 0.7985076962177716
 0.7985076962177716

Traits and custom measures

Notice that l1 reports per-sample evaluations, while rms only reports an aggregated result. This and other behavior can be gleaned from measure traits which are summarized by the info method:

julia> info(l1)
(target_scitype = Union{AbstractArray{Continuous,1}, AbstractArray{Count,1}},
 prediction_type = :deterministic,
 orientation = :loss,
 reports_each_observation = true,
 is_feature_dependent = false,
 supports_weights = true,)

A user-defined measure in MLJ can be passed to the evaluate! method, and elsewhere in MLJ, provided it is a function or callable object conforming to the above syntactic conventions. By default, a custom measure is understood to:

be a loss function (rather than a score or interval)
report an aggregated value (rather than per-sample evaluations)
be feature-independent

To override this behavior one simply overloads the appropriate trait, as shown in the following examples:

julia> y = [1, 2, 3, 4]; ŷ = [2, 3, 3, 3]; w = [1, 2, 2, 1];

julia> my_loss(ŷ, y) = maximum((ŷ - y).^2);

julia> my_loss(ŷ, y)
1

julia> my_per_sample_loss(ŷ, y) = abs.(ŷ - y);

julia> MLJ.reports_each_observation(::typeof(my_per_sample_loss)) = true;

julia> my_per_sample_loss(ŷ, y)
4-element Array{Int64,1}:
 1
 1
 0
 1

julia> my_weighted_score(ŷ, y) = 1/mean(abs.(ŷ - y));

julia> my_weighted_score(ŷ, y, w) = 1/mean(abs.((ŷ - y).^w));

julia> MLJ.supports_weights(::typeof(my_weighted_score)) = true;

julia> MLJ.orientation(::typeof(my_weighted_score)) = :score;

julia> my_weighted_score(ŷ, y)
1.3333333333333333

julia> X = (x=rand(4), penalty=[1, 2, 3, 4]);

julia> my_feature_dependent_loss(ŷ, X, y) = sum(abs.(ŷ - y) .* X.penalty)/sum(X.penalty);

julia> MLJ.is_feature_dependent(::typeof(my_feature_dependent_loss)) = true

julia> my_feature_dependent_loss(ŷ, X, y)
0.7

The possible signatures for custom measures are: measure(ŷ, y), measure(ŷ, y, w), measure(ŷ, X, y) and measure(ŷ, X, y, w), each measure implementing one non-weighted version, and possibly a second weighted version.

Implementation detail: Internally, every measure is evaluated using the syntax

MLJ.value(measure, ŷ, X, y, w)

and the traits determine what can be ignored and how measure is actually called. If w=nothing then the non-weighted form of measure is dipatched.

Using LossFunctions

The LossFunctions package includes "distance loss" functions for Continuous targets, and "marginal loss" functins for Binary targets. While the LossFunctions interface differs from the present one (for, example Binary observations must be +1 or -1), one can safely pass the loss functions defined there to any MLJ algorithm, which re-interprets it under the hood. Note that the distance loss functions apply to deterministic predictions, while the marginal losses apply to probabilistic predictions.

julia> using LossFunctions

julia> X = (x1=rand(5), x2=rand(5)); y = categorical(["y", "y", "y", "n", "y"]); w = [1, 2, 1, 2, 3];

julia> mach = machine(ConstantClassifier(), X, y);

julia> holdout = Holdout(fraction_train=0.6);

julia> evaluate!(mach,
                 measure=[ZeroOneLoss(), L1HingeLoss(), L2HingeLoss(), SigmoidLoss()],
                 resampling=holdout,
                 operation=predict,
                 weights=w,
                 verbosity=0)
(measure = LearnBase.MarginLoss[ZeroOneLoss(), L1HingeLoss(), L2HingeLoss(), SigmoidLoss()],
 measurement = [0.4, 0.8, 1.6, 0.847681],
 per_fold = Array{Float64,1}[[0.4], [0.8], [1.6], [0.847681]],
 per_observation = Array{Array{Float64,1},1}[[[0.8, 0.0]], [[1.6, 0.0]], [[3.2, 0.0]], [[1.40928, 0.286087]]],)

Note: Although one cannot directly call ZeroOneLoss() on categorical vectors, applying MLJ.value as discussed above has the expected behaviour:

julia> ŷ = predict(mach, X);

julia> MLJ.value(ZeroOneLoss(), ŷ, X, y, w) # X ignored here
5-element Array{Float64,1}:
 0.0
 0.0
 0.0
 1.1111111111111112
 0.0

julia> mean(MLJ.value(ZeroOneLoss(), ŷ, X, y, w)) ≈ misclassification_rate(ŷ, y, w)
false

API for built-in loss functions

MLJ.cross_entropy — Constant.

cross_entropy(ŷ, y::AbstractVector{<:Finite})

Given an abstract vector of UnivariateFinite distributions ŷ (ie, of probabilistic predictions) and an abstract vector of true observations y, return the negative log-probability that each observation would occur, according to the corresponding probabilistic prediction.

For more information, run info(cross_entropy).

source

MLJ.l1 — Constant.

l1(ŷ, y)
l1(ŷ, y, w)

L1 per-observation loss.

For more information, run info(l1).

source

MLJ.l2 — Constant.

l2(ŷ, y)
l2(ŷ, y, w)

L2 per-observation loss.

For more information, run info(l2).

source

MLJ.mav — Constant.

mav(ŷ, y)
mav(ŷ, y, w)

Mean absolute error (also known as MAE).

$\text{MAV} = n^{-1}∑ᵢ|yᵢ-ŷᵢ|$ or $\text{MAV} = ∑ᵢwᵢ|yᵢ-ŷᵢ|/∑ᵢwᵢ$

For more information, run info(mav).

source

MLJ.misclassification_rate — Constant.

misclassification_rate(ŷ, y)
misclassification_rate(ŷ, y, w)

Returns the rate of misclassification of the (point) predictions ŷ, given true observations y, optionally weighted by the weights w. All three arguments must be abstract vectors of the same length.

For more information, run info(misclassification_rate).

source

MLJ.rms — Constant.

rms(ŷ, y)
rms(ŷ, y, w)

Root mean squared error:

$\text{RMS} = \sqrt{n^{-1}∑ᵢ|yᵢ-ŷᵢ|^2}$ or $\text{RMS} = \sqrt{\frac{∑ᵢwᵢ|yᵢ-ŷᵢ|^2}{∑ᵢwᵢ}}$

For more information, run info(rms).

source

MLJ.rmsl — Constant.

rmsl(ŷ, y)

Root mean squared logarithmic error:

$\text{RMSL} = n^{-1}∑ᵢ\log\left({yᵢ \over ŷᵢ}\right)$

For more information, run info(rmsl).