# Tools

method | description |
---|---|

`measurements` `(measure, ...)` | for obtaining per-observation measurements, instead of aggregated ones |

`measures()` | dictionary of traits keyed on measure constructors, with filter options |

`unfussy(measure)` | new measure without argument checks¹ |

`multimeasure` `(measure; options...)` | wrapper to broadcast measures over multiple observations |

`robust_measure(measure)` | wrapper to silently treat unsupported weights as uniform |

`Measure(measure)` | wrapper for 3rd party measures with different calling syntax (e.g. LossFunctions.jl) |

¹For measures provided by StatisticalMeasures; behaviour for general measures may differ.

For more on defining your own measures, see the StatisticalMeasuresBase.jl documentation.

`StatisticalMeasuresBase.measurements`

— Function`measurements(measure, ŷ, y[, weights, class_weights::AbstractDict])`

Return a vector of measurements, one for each observation in `y`

, rather than a single aggregated measurement. Otherwise the behavior is the same as calling the measure directly on data.

**New implementations**

Overloading this function for new measure types is optional. A fallback returns the aggregated measure, repeated `n`

times, where `n = MLUtils.numobs(y)`

(which falls back to `length(y)`

if `numobs`

is not implemented). It is not typically necessary to overload `measurements`

for wrapped measures. All `multimeasure`

s provide the obvious fallback and other wrappers simply forward the `measurements`

method of the atomic measure. If overloading, use the following signatures:

```
StatisticalMeasuresBase.measurements(measure::SomeMeasureType, ŷ, y)
StatisticalMeasuresBase.measurements(measure::SomeMeasureType, ŷ, weights)
StatisticalMeasuresBase.measurements(measure::SomeMeasureType, ŷ, class_weights::AbstractDict)
StatisticalMeasuresBase.measurements(measure::SomeMeasureType, ŷ, weights, class_weights)
```

`StatisticalMeasures.measures`

— Function`measures(; trait_options...)`

*Experimental* and subject to breaking behavior between patch releases.

Return a dictionary, `dict`

, keyed on measure constructors provided by StatisticalMeasures.jl. The value of `dict[constructor]`

provides information about traits (measure "metadata") shared by all measures constructed using the syntax `constructor(args...)`

.

**Trait options**

One can filter on the basis of measure trait values, as shown in this example:

```
using StatisticalMeasures
import ScientificTypesBase.Multiclass
julia> measures(
observation_scitype = Union{Missing,Multiclass},
supports_class_weights = true,
)
```

```
measures(y; trait_filters...)
measures(yhat, y; trait_filters...)
```

*Experimental* and subject to breaking behavior between patch releases.

Assuming, ScientificTypes.jl has been imported, find measures that can be applied to data with the specified data arguments `(y,)`

or `(yhat, y)`

. It is assumed that the arguments contain multiple observations (have types implementing `MLUtils.getobs`

).

Returns a dictionary keyed on the constructors of such measures. Additional `trait_filters`

are the same as for the zero argument `measures`

method.

```
using StatisticalArrays
using ScientificTypes
julia> measures(rand(3), rand(3), supports_weights=false)
LittleDict{Any, Any, Vector{Any}, Vector{Any}} with 1 entry:
RSquared => (aliases = ("rsq", "rsquared"), consumes_multiple_observations = true, can_re…
```

*Warning.* Matching is based only on the *first* observation of the arguments provided, and must be interpreted carefully if, for example, `y`

or `yhat`

are vectors with `Union`

or other abstract element types.

`measures(needle::Union{AbstractString,Regex}; trait_options...)`

*Experimental* and subject to breaking behavior between patch releases.

Find measures that contain `needle`

in their document string. Returns a dictionary keyed on the constructors of such measures.

```
julia> measures("Matthew")
LittleDict{Any, Any, Vector{Any}, Vector{Any}} with 1 entry:
MatthewsCorrelation => (aliases = ("matthews_correlation", "mcc"), consumes_multiple_obse…
```

`StatisticalMeasuresBase.unfussy`

— Function`unfussy(measure)`

Return a version of `measure`

with argument checks removed, if that is possible. Specifically, if `measure == fussy_measure(atomic_measure)`

, for some `atomic_measure`

, then return `atomic_measure`

. Otherwise, return `measure`

.

See also `StatisticalMeasuresBase.fussy_measure`

.

`StatisticalMeasuresBase.multimeasure`

— Function`StatisticalMeasuresBase.multimeasure(atomic_measure; options...)`

Return a new measure, called a *multi-measure*, which, on a prediction-target pair `(ŷ, y)`

, broadcasts `atomic_measure`

over `MLUtils.eachobs((ŷ, y))`

and aggregates the result. Here `ŷ`

and `y`

are necessarily objects implementing the MLUtils `getobs/numobs`

interface, such as arrays, and tables `X`

for which `Tables.istable(X) == true`

.

All multi-measures automatically support weights and class weights.

By default, aggregation is performed using the preferred mode for `atomic_measure`

, i.e., `StatisticalMeasuresBase.external_aggregation_mode(atomic_measure)`

. Internally, aggregation is performed using the `aggregate`

method.

Nested applications of `multimeasure`

are useful for building measures that apply to matrices and some tables ("multi-targets") as well as multidimensional arrays. See the Advanced Examples below.

**Simple example**

```
using StatisticalMeasuresBase
# define an atomic measure:
struct L2OnScalars
(::L2OnScalars)(ŷ, y) = (ŷ - y)^2
julia> StatisticalMeasuresBase.external_aggregation_mode(L2OnScalars())
Mean()
# define a multimeasure:
L2OnVectors() = StatisticalMeasuresBase.multimeasure(L2OnScalars())
y = [1, 2, 3]
ŷ = [7, 6, 5]
@assert L2OnVectors()(ŷ, y) ≈ (ŷ - y).^2 |> mean
```

**Keyword options**

`mode=StatisticalMeasuresBase.external_aggregation_mode(atomic_measure)`

: mode for aggregating the results of broadcasting. Possible values include`Mean()`

and`Sum()`

. See`AggregationMode`

for all options and their meanings. Using`Mean()`

in conjunction with weights returns the usual weighted mean scaled by the average weight value. .`transform=identity`

: an optional transformation applied to observations in`y`

and`ŷ`

before passing to each`atomic_measure`

call. A useful value is`vec∘collect`

which is the identity on vectors, flattens arrays, and converts the observations of some tables (it's "rows") to vectors. See the example below.`atomic_weights=nothing`

: the weights to be passed to the atomic measure, on each call to evaluate it on the pair`(transform(ŷᵢ), transform(yᵢ))`

, for each`(ŷᵢ, yᵢ)`

in`MLUtils.eachjobs(ŷ, y)`

. Assumes`atomic_measure`

supports weights.`skipnan=false`

: whether to skip`NaN`

values when aggregating (`missing`

values are always skipped)

**Advanced examples**

Building on `L2OnVectors`

defined above:

```
# define measure for multi-dimensional arrays and some tables:
L2() = multimeasure(L2OnVectors(), transform=vec∘collect)
y = rand(3, 5, 100)
ŷ = rand(3, 5, 100)
weights = rand(100)
@assert L2()(ŷ, y, weights) ≈
sum(vec(mean((ŷ - y).^2, dims=[1, 2])).*weights)/length(weights)
using Tables
y = rand(3, 100)
ŷ = rand(3, 100)
t = Tables.table(y') |> Tables.rowtable
t̂ = Tables.table(ŷ') |> Tables.rowtable
@assert L2()(t̂, t, weights) ≈
sum(vec(mean((ŷ - y).^2, dims=1)).*weights)/length(weights)
```

The measure traits `StatisticalMeasuresBase.observation_scitype(measure)`

(default=`Union{}`

) and `StatisticalMeasuresBase.can_consume_tables(measure)`

(default=`false`

) are not forwarded from the atomic measure and must be explicitly overloaded for measures wrapped using `multimeasure`

.

`StatisticalMeasuresBase.robust_measure`

— Function`robust_measure(measure)`

Return a new measure `robust`

such that:

`weights`

and`class_weights`

are silently treated as uniform (unit) if unsupported by`measure`

if either

`weights`

or`class_weights`

is`nothing`

, it is as if the argument is omitted (interpreted as uniform)

This holds for all calls of the form `robust(ŷ, y, weights, class_weights)`

or `measurements(robust, ŷ, y, weights, class_weights)`

and otherwise the behavior of `robust`

is the same as for `measure`

.

`StatisticalMeasuresBase.Measure`

— Type`Measure(m)`

Convert a measure-like object `m`

to a measure in the sense of StatisticalMeasuresBase.jl; see `StatisticalMeasuresBase.is_measure`

for the definition.

Typically, `Measure`

is applied to measures with pre-existing calling behaviour different from that specified by StatisticalMeasuresBase.jl.

**New implementations**

To make a measure-like object of type `M`

wrappable by `Measure`

, implement the appropriate methods below. The first and last are compulsory.

```
(m::Measure{M})(ŷ, y)
(m::Measure{M})(ŷ, y, weights)
(m::Measure{M})(ŷ, y, class_weights::AbstractDict)
(m::Measure{M}, ŷ, y, weights, class_weights)
StatisticalMeasuresBase.measurements(m::Measure{M}, ŷ, y)
StatisticalMeasuresBase.measurements(m::Measure{M}, ŷ, y, weights)
StatisticalMeasuresBase.measurements(m::Measure{M}, ŷ, y, class_weights::AbstractDict)
StatisticalMeasuresBase.measurements(m::Measure{M}, ŷ, y, weights, class_weights)
StatisticalMeasuresBase.is_measure(m::Measure{M}) where M = true
```

In your implementations, you may use `StatisticalMeasuresBase.unwrap`

to access the unwrapped object, i.e., `StatisticalMeasuresBase.unwrap(Measure(m)) === m`

.

**Sample implementation**

To wrap the `abs`

function as a measure that computes the absolute value of differences:

```
import StatisticalMeasuresBase as API
(measure::API.Measure{typeof(abs)})(yhat, y) = API.unwrap(measure)(yhat - y)
API.is_measure(::API.Measure{typeof(abs)}) = true
julia> API.Measure(abs)(2, 5)
3
```