Credit Card Fraud

LogisticClassifier = @load LogisticClassifier pkg=MLJLinearModels
model_logit = LogisticClassifier(lambda=1.0)
mach = machine(model_logit, Xtrain, ytrain) |> fit!

import MLJLinearModels ✔
trained Machine; caches model-specific representations of data
  model: LogisticClassifier(lambda = 1.0, …)
  args: 
    1:	Source @030 ⏎ ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}}
    2:	Source @408 ⏎ AbstractVector{ScientificTypesBase.OrderedFactor{2}}

Predictions

LogisticClassifier is a probabilistic predictor, i.e. for each observation in the sample it attaches a probability to each of the possible values of the target. To recover a deterministic output, we use predict_mode instead of predict:

yhat_logit = predict_mode(mach, Xtest);
first(yhat_logit, 4)

# How does this model perform?

confusion_matrix(yhat_logit, ytest)

          ┌─────────────┐
          │Ground Truth │
┌─────────┼──────┬──────┤
│Predicted│  0   │  1   │
├─────────┼──────┼──────┤
│    0    │ 2843 │  5   │
├─────────┼──────┼──────┤
│    1    │  0   │  0   │
└─────────┴──────┴──────┘

To plot a receiver operator characteristic, we need the probabilistic predictions:

yhat = predict(mach, Xtest);
yhat[1:3]

3-element CategoricalDistributions.UnivariateFiniteVector{ScientificTypesBase.OrderedFactor{2}, Int64, UInt32, Float64}:
 UnivariateFinite{ScientificTypesBase.OrderedFactor{2}}(0=>0.998, 1=>0.00178)
 UnivariateFinite{ScientificTypesBase.OrderedFactor{2}}(0=>0.998, 1=>0.00174)
 UnivariateFinite{ScientificTypesBase.OrderedFactor{2}}(0=>0.998, 1=>0.00185)

false_positive_rates, true_positive_rates, thresholds =
    roc_curve(yhat, ytest)
plot(false_positive_rates, true_positive_rates)
plot!([0, 1], [0, 1], linewidth=2, linestyle=:dash, color=:black, label=:none)
xlabel!("false positive rate")
ylabel!("true positive rate")

misclassification_rate(yhat_logit, ytest)

0.0017556179775280898

Looks like it's not too bad. Let's see if we can do even better by doing a little tuning.

‎

‎

Still LogisticClassifier but implementing hyperparameter tuning.

r = range(model_logit, :lambda, lower=1e-6, upper=100, scale=:log)

self_tuning_logit_model = TunedModel(
    model_logit,
    tuning = Grid(resolution=10),
    resampling = CV(nfolds=3),
    range = r,
    measure = misclassification_rate,
)

mach = machine(self_tuning_logit_model, Xtrain, ytrain) |> fit!

trained Machine; does not cache data
  model: ProbabilisticTunedModel(model = LogisticClassifier(lambda = 1.0, …), …)
  args: 
    1:	Source @299 ⏎ ScientificTypesBase.Table{AbstractVector{ScientificTypesBase.Continuous}}
    2:	Source @426 ⏎ AbstractVector{ScientificTypesBase.OrderedFactor{2}}

Predictions

yhat_logit_tuned = predict_mode(mach, Xtest);

Let's take a look at the misclassification_rate. It is, surprisingly, slightly higher than the one calculated for the non tuned model.

@show misclassification_rate(yhat_logit_tuned, ytest)

misclassification_rate(yhat_logit_tuned, ytest) = 0.001053370786516854

This is lower, although the difference may not be statistically significant.

‎

‎

Initial SVM classification with cost = 1.0:

SVC = @load SVC pkg = LIBSVM

import MLJLIBSVMInterface ✔
MLJLIBSVMInterface.SVC

To fit the SVM, we declare a pipeline which comprises both a standardizer and the model. Training is substantially longer than for the preceding linear model (over 10 minutes):

model_svm = Standardizer() |>  SVC()
mach = machine(model_svm, Xtrain, ytrain) |> fit!
yhat_svm = predict(mach, Xtest)
confusion_matrix(yhat_svm, ytest)

          ┌─────────────┐
          │Ground Truth │
┌─────────┼──────┬──────┤
│Predicted│  0   │  1   │
├─────────┼──────┼──────┤
│    0    │ 2843 │  4   │
├─────────┼──────┼──────┤
│    1    │  0   │  1   │
└─────────┴──────┴──────┘

@show misclassification_rate(yhat_svm, ytest)

misclassification_rate(yhat_svm, ytest) = 0.0014044943820224719

Tuned SVM

r = range(model_svm, :(svc.cost), lower=0.1, upper=3.5, scale=:linear)
self_tuning_svm_model = TunedModel(
    model_svm,
    resampling = CV(nfolds=3),
    tuning = Grid(resolution=6),
    range = r,
    measure = misclassification_rate,
)
mach = machine(self_tuning_svm_model, Xtrain, ytrain) |> fit!

fitted_params(mach).best_model

DeterministicPipeline(
  standardizer = Standardizer(
        features = Symbol[], 
        ignore = false, 
        ordered_factor = false, 
        count = false), 
  svc = SVC(
        kernel = LIBSVM.Kernel.RadialBasis, 
        gamma = 0.0, 
        cost = 3.5, 
        cachesize = 200.0, 
        degree = 3, 
        coef0 = 0.0, 
        tolerance = 0.001, 
        shrinking = true), 
  cache = true)

plot(mach)

yhat_svm_tuned = predict(mach, Xtest)
confusion_matrix(yhat_svm_tuned, ytest)

          ┌─────────────┐
          │Ground Truth │
┌─────────┼──────┬──────┤
│Predicted│  0   │  1   │
├─────────┼──────┼──────┤
│    0    │ 2843 │  3   │
├─────────┼──────┼──────┤
│    1    │  0   │  2   │
└─────────┴──────┴──────┘

misclassification_rate(yhat_svm_tuned, ytest)

0.001053370786516854

‎

‎

NeuralNetworkClassifier = @load NeuralNetworkClassifier pkg=MLJFlux

import MLJFlux ✔
MLJFlux.NeuralNetworkClassifier

We assume familiarity with the building blocks of Flux models. In MLJFlux, a builder is essentially a rule for creating a Flux chain, once the data has been inspected for size. See the MLJFlux documentation for further details. We do note specify the softmax "finalizer" because MLJFlux classifiers add that under the hood.

import MLJFlux.@builder
using Flux

builder = @builder Chain(
    Dense(n_in, 16, relu),
    Dropout(0.1; rng=rng),
    Dense(16, n_out),
)

GenericBuilder(apply = #1)

In the @builder macro call, n_in, n_out, and rng are replaced with the actual number of input features found in the data, the actual number of output classes, and the rng specified in the model hyperparameters (see below).

We are now ready to specify the MLJFlux model. If you have running with GPU, you can try adding the option acceleration=CUDALibs().

rng = Xoshiro(123)
model = NeuralNetworkClassifier(
    ; builder,
    loss=(yhat, y)->Flux.tversky_loss(yhat, y, β=0.9), # combines precision and recall
    batch_size = round(Int, reduction*2048),
    epochs=50,
    rng,
)

NeuralNetworkClassifier(
  builder = GenericBuilder(
        apply = var"#1#2"()), 
  finaliser = NNlib.softmax, 
  optimiser = Flux.Optimise.Adam(0.001, (0.9, 0.999), 1.0e-8, IdDict{Any, Any}()), 
  loss = var"#3#4"(), 
  epochs = 50, 
  batch_size = 102, 
  lambda = 0.0, 
  alpha = 0.0, 
  rng = Random.Xoshiro(0xfefa8d41b8f5dca5, 0xf80cc98e147960c1, 0x20e2ccc17662fc1d, 0xea7a7dcb2e787c01, 0xf4e85a418b9c4f80), 
  optimiser_changes_trigger_retraining = false, 
  acceleration = ComputationalResources.CPU1{Nothing}(nothing))

Although we have not paid attention to it so far (and probably should have) there is substantial class imbalance for our target:

StatsBase.countmap(y)

Dict{CategoricalArrays.CategoricalValue{Int64, UInt32}, Int64} with 2 entries:
  0 => 284315
  1 => 492

We will address this by wrapping our model in a SMOTE overampling strategy, using MLJ's BalancedModel wrapper. Here are options for oversampling:

models("oversampler")

7-element Vector{NamedTuple{(:name, :package_name, :is_supervised, :abstract_type, :constructor, :deep_properties, :docstring, :fit_data_scitype, :human_name, :hyperparameter_ranges, :hyperparameter_types, :hyperparameters, :implemented_methods, :inverse_transform_scitype, :is_pure_julia, :is_wrapper, :iteration_parameter, :load_path, :package_license, :package_url, :package_uuid, :predict_scitype, :prediction_type, :reporting_operations, :reports_feature_importances, :supports_class_weights, :supports_online, :supports_training_losses, :supports_weights, :transform_scitype, :input_scitype, :target_scitype, :output_scitype)}}:
 (name = BorderlineSMOTE1, package_name = Imbalance, ... )
 (name = ROSE, package_name = Imbalance, ... )
 (name = RandomOversampler, package_name = Imbalance, ... )
 (name = RandomWalkOversampler, package_name = Imbalance, ... )
 (name = SMOTE, package_name = Imbalance, ... )
 (name = SMOTEN, package_name = Imbalance, ... )
 (name = SMOTENC, package_name = Imbalance, ... )

We'll use SMOTE:

SMOTE = @load SMOTE pkg=Imbalance
balanced_model = BalancedModel(model, oversampler=SMOTE())

import Imbalance ✔
BalancedModelProbabilistic(
  model = NeuralNetworkClassifier(
        builder = GenericBuilder(apply = #1), 
        finaliser = NNlib.softmax, 
        optimiser = Flux.Optimise.Adam(0.001, (0.9, 0.999), 1.0e-8, IdDict{Any, Any}()), 
        loss = var"#3#4"(), 
        epochs = 50, 
        batch_size = 102, 
        lambda = 0.0, 
        alpha = 0.0, 
        rng = Random.Xoshiro(0xfefa8d41b8f5dca5, 0xf80cc98e147960c1, 0x20e2ccc17662fc1d, 0xea7a7dcb2e787c01, 0xf4e85a418b9c4f80), 
        optimiser_changes_trigger_retraining = false, 
        acceleration = ComputationalResources.CPU1{Nothing}(nothing)), 
  oversampler = SMOTE(
        k = 5, 
        ratios = 1.0, 
        rng = Random.TaskLocalRNG(), 
        try_preserve_type = true))

Our final model adds standarization as a pre-processor:

model_nn = Standardizer() |> balanced_model

ProbabilisticPipeline(
  standardizer = Standardizer(
        features = Symbol[], 
        ignore = false, 
        ordered_factor = false, 
        count = false), 
  balanced_model_probabilistic = BalancedModelProbabilistic(
        model = NeuralNetworkClassifier(builder = GenericBuilder(apply = #1), …), 
        oversampler = SMOTE(k = 5, …)), 
  cache = true)

mach = machine(model_nn, Xtrain, ytrain) |> fit!
yhat_nn = predict_mode(mach, Xtest);
confusion_matrix(yhat_nn, ytest)

          ┌─────────────┐
          │Ground Truth │
┌─────────┼──────┬──────┤
│Predicted│  0   │  1   │
├─────────┼──────┼──────┤
│    0    │ 2843 │  2   │
├─────────┼──────┼──────┤
│    1    │  0   │  3   │
└─────────┴──────┴──────┘

misclassification_rate(yhat_nn, ytest)

0.0007022471910112359

‎