GitHub

Confusion Matrices

Users typically construct confusion matrices using the confmat measure, or other variants that can be constructed using the ConfusionMatrix constructor. See Examples of Usage for examples. (A pure function version of confmat, with options, is ConfusionMatrices.confmat.)

The ConfusionMatrices submodule of StatisticalMeasures.jl provides some methods for extracting data from these matrices, detailed below.

methoddescription
cm[i, j]count for a ith class prediction and jth class ground truth
cm(p, g)count for a class p prediction and class g ground truth
ConfusionMatrices.matrix(cm)return the raw matrix associated with confusion matrix cm
ConfusionMatrices.isordered(cm)true if levels of cm have been explicitly ordered
ConfusionMatrices.levels(cm)return the target levels (classes) associated with cm

Reference

CategoricalArrays.isordered — Function
ConfusionMatrices.isordered(m::ConfusionMatrix)

Return true if and only if the levels associated with m have been explicitly ordered.

source
DataAPI.levels — Function
levels(m::ConfusionMatrices.ConfusionMatrix)

Return the levels associated with the confusion matrix m, in the order consistent with the regular matrix returned by ConfusionMatrices.matrix(cm).

!!! note New in StatisticalMeasures 0.3

For confusion matrices constructed using `CategoricalArray`s, a `CategoricalVector` is
returned. Previously the levels were unwrapped, to match the old behavior of
`CategoricalArrays.levels`.
source
StatisticalMeasures.ConfusionMatrices.ConfusionMatrix — Type
ConfusionMatrices.ConfusionMatrix{N,O,L}

Wrapper type for confusion matrices.

Type parameters

  • N ≥ 2: number of classes (levels)

  • O: true if levels are explicitly understood to be ordered

  • L: type of labels

Predicted classes (levels) are constant on rows, ground truth classes are constant on columns.

See the Confusion matrix wikipedia article for more information.

Public interface

Instances can be constructed directly using the ConfusionMatrix constructor (two methods documented below) or, more typically, using ConfusionMatrices.confmat. Other methods are: ConfusionMatrices.matrix (to extract raw matrix), levels, and isordered.

Two instances are considered == if:

  • The associated levels agree, as sets

  • If both instances are ordered, then the levels also agree as vectors

  • Access-by-level behaviour is the same (see below)

Instances need not have the same underlying matrix to be ==.

Access elements via level as shown in this example:

import StatisticalMeasures.ConfusionMatrices as CM

y = ["a", "b", "a", "a", "b", "a", "a", "b", "b", "a"]
ŷ = ["b", "a", "a", "b", "a", "b", "b", "b", "a", "a"]

julia> cm = CM.confmat(ŷ, y)
              ┌───────────────────────────┐
              │       Ground Truth        │
┌─────────────┼─────────────┬─────────────┤
│  Predicted  │      a      │      b      │
├─────────────┼─────────────┼─────────────┤
│      a      │      2      │      3      │
├─────────────┼─────────────┼─────────────┤
│      b      │      4      │      1      │
└─────────────┴─────────────┴─────────────┘


julia> cm("a", "b")
3

Access by index is also possible, if the confusion matrix is ordered. Otherwise, you can first extract the underlying matrix with ConfusionMatrices.matrix. For options creating ordered confusion matrices, see ConfusionMatrices.confmat.

source
StatisticalMeasures.ConfusionMatrices.confmat — Function
ConfusionMatrices.confmat(ŷ, y, levels=nothing, rev=false, perm=nothing, checks=true)

Return the confusion matrix corresponding to predictions ŷ and ground truth observations y. Whenever missing occurs the corresponding prediction-ground-truth pair is skipped in the counting.

Elements of a confusion matrix can always be accessed by level - see the example below. To flag the confusion matrix as ordered, and hence index-accessible, do one of the following:

  • Supply ordered CategoricalArray inputs ŷ and y

  • Explicitly specify levels or one of rev, perm

Note that == for two confusion matrices is stricter when both are ordered.

Method is optimized for CategoricalArray inputs with levels inferred. In that case levels will be the complete internal class pool, and not just the observed levels.

import StatisticalMeasures.ConfusionMatrices as CM

y = ["a", "b", "a", "a", "b", "a", "a", "b", "b", "a"]
ŷ = ["b", "a", "a", "b", "a", "b", "b", "b", "a", "a"]

julia> cm = CM.confmat(ŷ, y)
              ┌───────────────────────────┐
              │       Ground Truth        │
┌─────────────┼─────────────┬─────────────┤
│  Predicted  │      a      │      b      │
├─────────────┼─────────────┼─────────────┤
│      a      │      2      │      3      │
├─────────────┼─────────────┼─────────────┤
│      b      │      4      │      1      │
└─────────────┴─────────────┴─────────────┘


julia> cm("a", "b")
3

julia> CM.matrix(cm)
┌ Warning: Confusion matrix levels not explicitly ordered. Using the order, ["a", "b"].
└ @ StatisticalMeasures.ConfusionMatrices ~/MLJ/StatisticalMeasures/src/confusion_matrices.jl:120
2×2 Matrix{Int64}:
 2  3
 4  1

ordered_cm = CM.confmat(ŷ, y, levels=["b", "a"])

julia> ordered_cm("a", "b")
3

julia> CM.matrix(ordered_cm)
2×2 Matrix{Int64}:
 1  4
 3  2

julia> ordered_cm[2, 1]
3

Keyword options

  • levels::Union{Vector,Nothing}=nothing: if nothing, levels are inferred from ŷ and y and, by default, ordered according to the element type of y.

  • rev=false: in the case of binary data, whether to reverse the levels (as inferred or specified); a nothing value is the same as false.

  • perm=nothing: in the general case, a permutation representing a re-ordering of levels (as inferred or specified); e.g., perm = [1,3,2] for data with three classes (levels).
  • checks=true: when true, specified levels are checked to see they include all observed levels; set to false for speed.

See also ConfusionMatrices.ConfusionMatrix, and the Confusion matrix wikipedia article.

source