Tools

measurements(measure, ŷ, y[, weights, class_weights::AbstractDict])

Return a vector of measurements, one for each observation in y, rather than a single aggregated measurement. Otherwise the behavior is the same as calling the measure directly on data.

New implementations

Overloading this function for new measure types is optional. A fallback returns the aggregated measure, repeated n times, where n = MLUtils.numobs(y) (which falls back to length(y) if numobs is not implemented). It is not typically necessary to overload measurements for wrapped measures. All multimeasures provide the obvious fallback and other wrappers simply forward the measurements method of the atomic measure. If overloading, use the following signatures:

StatisticalMeasuresBase.measurements(measure::SomeMeasureType, ŷ, y)
StatisticalMeasuresBase.measurements(measure::SomeMeasureType, ŷ, weights)
StatisticalMeasuresBase.measurements(measure::SomeMeasureType, ŷ, class_weights::AbstractDict)
StatisticalMeasuresBase.measurements(measure::SomeMeasureType, ŷ, weights, class_weights)

measures(; trait_options...)

Experimental and subject to breaking behavior between patch releases.

Return a dictionary, dict, keyed on measure constructors provided by StatisticalMeasures.jl. The value of dict[constructor] provides information about traits (measure "metadata") shared by all measures constructed using the syntax constructor(args...).

Trait options

One can filter on the basis of measure trait values, as shown in this example:

using StatisticalMeasures
import ScientificTypesBase.Multiclass

julia> measures(
    observation_scitype = Union{Missing,Multiclass},
    supports_class_weights = true,
)

measures(y; trait_filters...)
measures(yhat, y; trait_filters...)

Experimental and subject to breaking behavior between patch releases.

Assuming, ScientificTypes.jl has been imported, find measures that can be applied to data with the specified data arguments (y,) or (yhat, y). It is assumed that the arguments contain multiple observations (have types implementing MLUtils.getobs).

Returns a dictionary keyed on the constructors of such measures. Additional trait_filters are the same as for the zero argument measures method.

using StatisticalArrays
using ScientificTypes

julia> measures(rand(3), rand(3), supports_weights=false)
LittleDict{Any, Any, Vector{Any}, Vector{Any}} with 1 entry:
  RSquared => (aliases = ("rsq", "rsquared"), consumes_multiple_observations = true, can_re…

Warning. Matching is based only on the first observation of the arguments provided, and must be interpreted carefully if, for example, y or yhat are vectors with Union or other abstract element types.

measures(needle::Union{AbstractString,Regex}; trait_options...)

Experimental and subject to breaking behavior between patch releases.

Find measures that contain needle in their document string. Returns a dictionary keyed on the constructors of such measures.

julia> measures("Matthew")
LittleDict{Any, Any, Vector{Any}, Vector{Any}} with 1 entry:
  MatthewsCorrelation => (aliases = ("matthews_correlation", "mcc"), consumes_multiple_obse…

unfussy(measure)

Return a version of measure with argument checks removed, if that is possible. Specifically, if measure == fussy_measure(atomic_measure), for some atomic_measure, then return atomic_measure. Otherwise, return measure.

See also StatisticalMeasuresBase.fussy_measure.

StatisticalMeasuresBase.multimeasure(atomic_measure; options...)

Return a new measure, called a multi-measure, which, on a prediction-target pair (ŷ, y), broadcasts atomic_measure over MLUtils.eachobs((ŷ, y)) and aggregates the result. Here ŷ and y are necessarily objects implementing the MLUtils getobs/numobs interface, such as arrays, and tables X for which Tables.istable(X) == true.

All multi-measures automatically support weights and class weights.

By default, aggregation is performed using the preferred mode for atomic_measure, i.e., StatisticalMeasuresBase.external_aggregation_mode(atomic_measure). Internally, aggregation is performed using the aggregate method.

Nested applications of multimeasure are useful for building measures that apply to matrices and some tables ("multi-targets") as well as multidimensional arrays. See the Advanced Examples below.

Simple example

using StatisticalMeasuresBase

# define an atomic measure:
struct L2OnScalars
(::L2OnScalars)(ŷ, y) = (ŷ - y)^2

julia> StatisticalMeasuresBase.external_aggregation_mode(L2OnScalars())
Mean()

# define a multimeasure:
L2OnVectors() = StatisticalMeasuresBase.multimeasure(L2OnScalars())

y = [1, 2, 3]
ŷ = [7, 6, 5]
@assert L2OnVectors()(ŷ, y) ≈ (ŷ - y).^2 |> mean

Keyword options

• mode=StatisticalMeasuresBase.external_aggregation_mode(atomic_measure): mode for aggregating the results of broadcasting. Possible values include Mean() and Sum(). See AggregationMode for all options and their meanings. Using Mean() in conjunction with weights returns the usual weighted mean scaled by the average weight value. .

• transform=identity: an optional transformation applied to observations in y and ŷ before passing to each atomic_measure call. A useful value is vec∘collect which is the identity on vectors, flattens arrays, and converts the observations of some tables (it's "rows") to vectors. See the example below.

• atomic_weights=nothing: the weights to be passed to the atomic measure, on each call to evaluate it on the pair (transform(ŷᵢ), transform(yᵢ)), for each (ŷᵢ, yᵢ) in MLUtils.eachjobs(ŷ, y). Assumes atomic_measure supports weights.

• skipnan=false: whether to skip NaN values when aggregating (missing values are always skipped)

Advanced examples

Building on L2OnVectors defined above:

# define measure for multi-dimensional arrays and some tables:
L2() = multimeasure(L2OnVectors(), transform=vec∘collect)

y = rand(3, 5, 100)
ŷ = rand(3, 5, 100)
weights = rand(100)
@assert L2()(ŷ, y, weights) ≈
   sum(vec(mean((ŷ - y).^2, dims=[1, 2])).*weights)/length(weights)

using Tables
y = rand(3, 100)
ŷ = rand(3, 100)
t = Tables.table(y') |> Tables.rowtable
t̂ = Tables.table(ŷ') |> Tables.rowtable
@assert L2()(t̂, t, weights) ≈
   sum(vec(mean((ŷ - y).^2, dims=1)).*weights)/length(weights)

robust_measure(measure)

Return a new measure robust such that:

• weights and class_weights are silently treated as uniform (unit) if unsupported by measure

• if either weights or class_weights is nothing, it is as if the argument is omitted (interpreted as uniform)

This holds for all calls of the form robust(ŷ, y, weights, class_weights) or measurements(robust, ŷ, y, weights, class_weights) and otherwise the behavior of robust is the same as for measure.

Measure(m)

Convert a measure-like object m to a measure in the sense of StatisticalMeasuresBase.jl; see StatisticalMeasuresBase.is_measure for the definition.

Typically, Measure is applied to measures with pre-existing calling behaviour different from that specified by StatisticalMeasuresBase.jl.

New implementations

To make a measure-like object of type M wrappable by Measure, implement the appropriate methods below. The first and last are compulsory.

(m::Measure{M})(ŷ, y)
(m::Measure{M})(ŷ, y, weights)
(m::Measure{M})(ŷ, y, class_weights::AbstractDict)
(m::Measure{M}, ŷ, y, weights, class_weights)
StatisticalMeasuresBase.measurements(m::Measure{M}, ŷ, y)
StatisticalMeasuresBase.measurements(m::Measure{M}, ŷ, y, weights)
StatisticalMeasuresBase.measurements(m::Measure{M}, ŷ, y, class_weights::AbstractDict)
StatisticalMeasuresBase.measurements(m::Measure{M}, ŷ, y, weights, class_weights)
StatisticalMeasuresBase.is_measure(m::Measure{M}) where M = true

In your implementations, you may use StatisticalMeasuresBase.unwrap to access the unwrapped object, i.e., StatisticalMeasuresBase.unwrap(Measure(m)) === m.

Sample implementation

To wrap the abs function as a measure that computes the absolute value of differences:

import StatisticalMeasuresBase as API

(measure::API.Measure{typeof(abs)})(yhat, y) = API.unwrap(measure)(yhat - y)
API.is_measure(::API.Measure{typeof(abs)}) = true

julia> API.Measure(abs)(2, 5)
3