Confusion Matrices

Users typically construct confusion matrices using the confmat measure, or other variants that can be constructed using the ConfusionMatrix constructor. See Examples of Usage for examples. (A pure function version of confmat, with options, is ConfusionMatrices.confmat.)

The ConfusionMatrices submodule of StatisticalMeasures.jl provides some methods for extracting data from these matrices, detailed below.

method	description
`cm[i, j]`	count for a `i`th class prediction and `j`th class ground truth
`cm(p, g)`	count for a class `p` prediction and class `g` ground truth
`ConfusionMatrices.matrix(cm)`	return the raw matrix associated with confusion matrix `cm`
`ConfusionMatrices.isordered(cm)`	`true` if levels of `cm` have been explicitly ordered
`ConfusionMatrices.levels(cm)`	return the target levels (classes) associated with `cm`

Reference

StatisticalMeasures.ConfusionMatrices.matrix — Function

ConfusionMatrices.matrix(m::ConfusionMatrix; warn=true)

Return the regular Matrix associated with confusion matrix m.

source

CategoricalArrays.isordered — Function

ConfusionMatrices.isordered(m::ConfusionMatrix)

Return true if and only if the levels associated with m have been explicitly ordered.

source

DataAPI.levels — Function

levels(m::ConfusionMatrices.ConfusionMatrix)

Return the levels associated with the confusion matrix m, in the order consistent with the regular matrix returned by ConfusionMatrices.matrix(cm).

source

StatisticalMeasures.ConfusionMatrices.ConfusionMatrix — Type

ConfusionMatrices.ConfusionMatrix{N,O,L}

Wrapper type for confusion matrices.

Type parameters

N ≥ 2: number of levels (classes)
O: true if levels are explicitly understood to be ordered
L: type of labels

Predicted classes are constant on rows, ground truth classes are constant on columns.

See the Confusion matrix wikipedia article for more information.

Public interface

Instances can be constructed directly using the ConfusionMatrix constructor (two methods documented below) or, more typically, using ConfusionMatrices.confmat. Other methods are: ConfusionMatrices.matrix (to extract raw matrix), levels, and isordered.

Two instances are considered == if:

The associated levels agree, as sets
If both instances are ordered, then the levels also agree as vectors
Access-by-level behaviour is the same (see below)

Instances need not have the same underlying matrix to be ==.

Access elements via level as shown in this example:

import StatisticalMeasures.ConfusionMatrices as CM

y = ["a", "b", "a", "a", "b", "a", "a", "b", "b", "a"]
ŷ = ["b", "a", "a", "b", "a", "b", "b", "b", "a", "a"]

julia> cm = CM.confmat(ŷ, y)
              ┌───────────────────────────┐
              │       Ground Truth        │
┌─────────────┼─────────────┬─────────────┤
│  Predicted  │      a      │      b      │
├─────────────┼─────────────┼─────────────┤
│      a      │      2      │      3      │
├─────────────┼─────────────┼─────────────┤
│      b      │      4      │      1      │
└─────────────┴─────────────┴─────────────┘


julia> cm("a", "b")
3

Access by index is also possible, if the confusion matrix is ordered. Otherwise, you can first extract the underlying matrix with ConfusionMatrices.matrix. For options creating ordered confusion matrices, see ConfusionMatrices.confmat.

source

StatisticalMeasures.ConfusionMatrices.confmat — Function

ConfusionMatrices.confmat(ŷ, y, levels=nothing, rev=false, perm=nothing, checks=true)

Return the confusion matrix corresponding to predictions ŷ and ground truth observations y. Whenever missing occurs the corresponding prediction-ground-truth pair is skipped in the counting.

Elements of a confusion matrix can always be accessed by level - see the example below. To flag the confusion matrix as ordered, and hence index-accessible, do one of the following:

Supply ordered CategoricalArray inputs ŷ and y
Explicitly specify levels or one of rev, perm

Note that == for two confusion matrices is stricter when both are ordered.

Method is optimized for CategoricalArray inputs with levels inferred. In that case levels will be the complete internal class pool, and not just the observed levels.

import StatisticalMeasures.ConfusionMatrices as CM

y = ["a", "b", "a", "a", "b", "a", "a", "b", "b", "a"]
ŷ = ["b", "a", "a", "b", "a", "b", "b", "b", "a", "a"]

julia> cm = CM.confmat(ŷ, y)
              ┌───────────────────────────┐
              │       Ground Truth        │
┌─────────────┼─────────────┬─────────────┤
│  Predicted  │      a      │      b      │
├─────────────┼─────────────┼─────────────┤
│      a      │      2      │      3      │
├─────────────┼─────────────┼─────────────┤
│      b      │      4      │      1      │
└─────────────┴─────────────┴─────────────┘


julia> cm("a", "b")
3

julia> CM.matrix(cm)
┌ Warning: Confusion matrix levels not explicitly ordered. Using the order, ["a", "b"].
└ @ StatisticalMeasures.ConfusionMatrices ~/MLJ/StatisticalMeasures/src/confusion_matrices.jl:120
2×2 Matrix{Int64}:
 2  3
 4  1

ordered_cm = CM.confmat(ŷ, y, levels=["b", "a"])

julia> ordered_cm("a", "b")
3

julia> CM.matrix(ordered_cm)
2×2 Matrix{Int64}:
 1  4
 3  2

julia> ordered_cm[2, 1]
3

Keyword options

levels::Union{Vector,Nothing}=nothing: if nothing, levels are inferred from ŷ and y and, by default, ordered according to the element type of y.
rev=false: in the case of binary data, whether to reverse the levels (as inferred or specified); a nothing value is the same as false.

perm=nothing: in the general case, a permutation representing a re-ordering of levels (as inferred or specified); e.g., perm = [1,3,2] for data with three classes.

checks=true: when true, specified levels are checked to see they include all observed levels; set to false for speed.

source