Measures

ConfusionMatrixObject{C}

Confusion matrix with C ≥ 2 classes. Rows correspond to predicted values and columns to the ground truth.

ConfusionMatrixObject(m, labels)

Instantiates a confusion matrix out of a square integer matrix m. Rows are the predicted class, columns the ground truth. See also the wikipedia article.

_confmat(ŷ, y; rev=false)

A private method. General users should use confmat or other instance of the measure type ConfusionMatrix.

Computes the confusion matrix given a predicted ŷ with categorical elements and the actual y. Rows are the predicted class, columns the ground truth. The ordering follows that of levels(y).

Keywords

• rev=false: in the binary case, this keyword allows to swap the ordering of classes.
• perm=[]: in the general case, this keyword allows to specify a permutation re-ordering the classes.
• warn=true: whether to show a warning in case y does not have scientific type OrderedFactor{2} (see note below).

Note

To decrease the risk of unexpected errors, if y does not have scientific type OrderedFactor{2} (and so does not have a "natural ordering" negative-positive), a warning is shown indicating the current order unless the user explicitly specifies either rev or perm in which case it's assumed the user is aware of the class ordering.

The confusion_matrix is a measure (although neither a score nor a loss) and so may be specified as such in calls to evaluate, evaluate!, although not in TunedModels. In this case, however, there no way to specify an ordering different from levels(y), where y is the target.

MulticlassFScore(; β=1.0, average=macro_avg, return_type=LittleDict)

One-parameter generalization, $F_β$, of the F-measure or balanced F-score for multiclass observations.

MulticlassFScore()(ŷ, y)
MulticlassFScore()(ŷ, y, class_w)

Evaluate the default score on multiclass observations, ŷ, given ground truth values, y. Options for average are: no_avg, macro_avg (default) and micro_avg. Options for return_type, applying in the no_avg case, are: LittleDict (default) or Vector. An optional AbstractDict, denoted class_w above, keyed on levels(y), specifies class weights. It applies if average=macro_avg or average=no_avg.

For more information, run info(MulticlassFScore).

MulticlassFalseDiscoveryRate(; average=macro_avg, return_type=LittleDict)

multiclass false discovery rate; aliases: multiclass_false_discovery_rate, multiclass_falsediscovery_rate, multiclass_fdr.

MulticlassFalseDiscoveryRate()(ŷ, y)
MulticlassFalseDiscoveryRate()(ŷ, y, class_w)

False discovery rate for multiclass observations ŷ and ground truth y, using default averaging and return type. Options for average are: no_avg, macro_avg (default) and micro_avg. Options for return_type, applying in the no_avg case, are: LittleDict (default) or Vector. An optional AbstractDict, denoted class_w above, keyed on levels(y), specifies class weights. It applies if average=macro_avg or average=no_avg.

For more information, run info(MulticlassFalseDiscoveryRate).

MulticlassFalseNegativeRate(; average=macro_avg, return_type=LittleDict)

multiclass false negative rate; aliases: multiclass_false_negative_rate, multiclass_fnr, multiclass_miss_rate, multiclass_falsenegative_rate.

MulticlassFalseNegativeRate()(ŷ, y)
MulticlassFalseNegativeRate()(ŷ, y, class_w)

False negative rate for multiclass observations ŷ and ground truth y, using default averaging and return type. Options for average are: no_avg, macro_avg (default) and micro_avg. Options for return_type, applying in the no_avg case, are: LittleDict (default) or Vector. An optional AbstractDict, denoted class_w above, keyed on levels(y), specifies class weights. It applies if average=macro_avg or average=no_avg.

For more information, run info(MulticlassFalseNegativeRate).

MulticlassFalsePositiveRate(; average=macro_avg, return_type=LittleDict)

multiclass false positive rate; aliases: multiclass_false_positive_rate, multiclass_fpr multiclass_fallout, multiclass_falsepositive_rate.

MulticlassFalsePositiveRate()(ŷ, y)
MulticlassFalsePositiveRate()(ŷ, y, class_w)

False positive rate for multiclass observations ŷ and ground truth y, using default averaging and return type. Options for average are: no_avg, macro_avg (default) and micro_avg. Options for return_type, applying in the no_avg case, are: LittleDict (default) or Vector. An optional AbstractDict, denoted class_w above, keyed on levels(y), specifies class weights. It applies if average=macro_avg or average=no_avg.

For more information, run info(MulticlassFalsePositiveRate).

MulticlassNegativePredictiveValue(; average=macro_avg, return_type=LittleDict)

multiclass negative predictive value; aliases: multiclass_negative_predictive_value, multiclass_negativepredictive_value, multiclass_npv.

MulticlassNegativePredictiveValue()(ŷ, y)
MulticlassNegativePredictiveValue()(ŷ, y, class_w)

Negative predictive value for multiclass observations ŷ and ground truth y, using default averaging and return type. Options for average are: no_avg, macro_avg (default) and micro_avg. Options for return_type, applying in the no_avg case, are: LittleDict (default) or Vector. An optional AbstractDict, denoted class_w above, keyed on levels(y), specifies class weights. It applies if average=macro_avg or average=no_avg.

For more information, run info(MulticlassNegativePredictiveValue).

MulticlassPrecision(; average=macro_avg, return_type=LittleDict)

multiclass positive predictive value (aka precision); aliases: multiclass_positive_predictive_value, multiclass_ppv, multiclass_positivepredictive_value, multiclass_recall.

MulticlassPrecision()(ŷ, y)
MulticlassPrecision()(ŷ, y, class_w)

Precision for multiclass observations ŷ and ground truth y, using default averaging and return type. Options for average are: no_avg, macro_avg (default) and micro_avg. Options for return_type, applying in the no_avg case, are: LittleDict (default) or Vector. An optional AbstractDict, denoted class_w above, keyed on levels(y), specifies class weights. It applies if average=macro_avg or average=no_avg.

For more information, run info(MulticlassPrecision).

MulticlassTrueNegative(; return_type=LittleDict)

Number of true negatives; aliases: multiclass_true_negative, multiclass_truenegative.

MulticlassTrueNegative()(ŷ, y)

Number of true negatives for multiclass observations ŷ and ground truth y, using default return type. Options for return_type are: LittleDict(default) or Vector.

For more information, run info(MulticlassTrueNegative).

MulticlassTrueNegativeRate(; average=macro_avg, return_type=LittleDict)

multiclass true negative rate; aliases: multiclass_true_negative_rate, multiclass_tnr multiclass_specificity, multiclass_selectivity, multiclass_truenegative_rate.

MulticlassTrueNegativeRate()(ŷ, y)
MulticlassTrueNegativeRate()(ŷ, y, class_w)

True negative rate for multiclass observations ŷ and ground truth y, using default averaging and return type. Options for average are: no_avg, macro_avg (default) and micro_avg. Options for return_type, applying in the no_avg case, are: LittleDict (default) or Vector. An optional AbstractDict, denoted class_w above, keyed on levels(y), specifies class weights. It applies if average=macro_avg or average=no_avg.

For more information, run info(MulticlassTrueNegativeRate).

MulticlassTruePositive(; return_type=LittleDict)

Number of true positives; aliases: multiclass_true_positive, multiclass_truepositive.

MulticlassTruePositive()(ŷ, y)

Number of true positives for multiclass observations ŷ and ground truth y, using default return type. Options for return_type are: LittleDict(default) or Vector.

For more information, run info(MulticlassTruePositive).

MulticlassTruePositiveRate(; average=macro_avg, return_type=LittleDict)

multiclass true positive rate; aliases: multiclass_true_positive_rate, multiclass_tpr, multiclass_sensitivity, multiclass_recall, multiclass_hit_rate, multiclass_truepositive_rate,

MulticlassTruePositiveRate(ŷ, y)
MulticlassTruePositiveRate(ŷ, y, class_w)

True positive rate (a.k.a. sensitivity, recall, hit rate) for multiclass observations ŷ and ground truth y, using default averaging and return type. Options for average are: no_avg, macro_avg (default) and micro_avg. Options for return_type, applying in the no_avg case, are: LittleDict (default) or Vector. An optional AbstractDict, denoted class_w above, keyed on levels(y), specifies class weights. It applies if average=macro_avg or average=no_avg.

For more information, run info(MulticlassTruePositiveRate).

fprs, tprs, ts = roc_curve(ŷ, y) = roc(ŷ, y)

Return the ROC curve for a two-class probabilistic prediction ŷ given the ground truth y. The true positive rates, false positive rates over a range of thresholds ts are returned. Note that if there are k unique scores, there are correspondingly k thresholds and k+1 "bins" over which the FPR and TPR are constant:

• [0.0 - thresh[1]]
• [thresh[1] - thresh[2]]
• ...
• [thresh[k] - 1]

consequently, tprs and fprs are of length k+1 if ts is of length k.

To draw the curve using your favorite plotting backend, do plot(fprs, tprs).

.

.

MulticlassFalseNegative(; return_type=LittleDict)

Number of false negatives; aliases: multiclass_false_negative, multiclass_falsenegative.

MulticlassFalseNegative()(ŷ, y)

Number of false negatives for multiclass observations ŷ and ground truth y, using default return type. Options for return_type are: LittleDict(default) or Vector.

For more information, run info(MulticlassFalseNegative).

MulticlassFalsePositive(; return_type=LittleDict)

Number of false positives; aliases: multiclass_false_positive, multiclass_falsepositive.

MulticlassFalsePositive()(ŷ, y)

Number of false positives for multiclass observations ŷ and ground truth y, using default return type. Options for return_type are: LittleDict(default) or Vector.

For more information, run info(MulticlassFalsePositive).

_idx_unique_sorted(v)

Internal function to return the index of unique elements in v under the assumption that the vector v is sorted in decreasing order.