Common MLJ Workflows

This demo assumes you have certain packages in your active package environment. To activate a new environment, "MyNewEnv", with just these packages, do this in a new REPL session:

using Pkg
Pkg.activate("MyNewEnv")
Pkg.add(["MLJ", "RDatasets", "DataFrames", "MLJDecisionTreeInterface",
    "MLJMultivariateStatsInterface", "NearestNeighborModels", "MLJGLMInterface",
    "Plots"])

The following starts MLJ and shows the current version of MLJ (you can also use Pkg.status()):

using MLJ
MLJ_VERSION
v"0.20.7"

Data ingestion

import RDatasets
channing = RDatasets.dataset("boot", "channing")
first(channing, 4) |> pretty
┌──────────────────────────────────┬───────┬───────┬───────┬───────┐
│ Sex                              │ Entry │ Exit  │ Time  │ Cens  │
│ CategoricalValue{String, UInt32} │ Int32 │ Int32 │ Int32 │ Int32 │
│ Multiclass{2}                    │ Count │ Count │ Count │ Count │
├──────────────────────────────────┼───────┼───────┼───────┼───────┤
│ Male                             │ 782   │ 909   │ 127   │ 1     │
│ Male                             │ 1020  │ 1128  │ 108   │ 1     │
│ Male                             │ 856   │ 969   │ 113   │ 1     │
│ Male                             │ 915   │ 957   │ 42    │ 1     │
└──────────────────────────────────┴───────┴───────┴───────┴───────┘

Inspecting metadata, including column scientific types:

schema(channing)
┌───────┬───────────────┬──────────────────────────────────┐
│ names │ scitypes      │ types                            │
├───────┼───────────────┼──────────────────────────────────┤
│ Sex   │ Multiclass{2} │ CategoricalValue{String, UInt32} │
│ Entry │ Count         │ Int32                            │
│ Exit  │ Count         │ Int32                            │
│ Time  │ Count         │ Int32                            │
│ Cens  │ Count         │ Int32                            │
└───────┴───────────────┴──────────────────────────────────┘

Horizontally splitting data and shuffling rows.

Here y is the :Exit column and X a table with everything else:

y, X = unpack(channing, ==(:Exit), rng=123)

Here y is the :Exit column and X everything else except :Time:

y, X = unpack(channing,
              ==(:Exit),
              !=(:Time);
              rng=123);
scitype(y)
AbstractVector{Count} (alias for AbstractArray{Count, 1})
schema(X)
┌───────┬───────────────┬──────────────────────────────────┐
│ names │ scitypes      │ types                            │
├───────┼───────────────┼──────────────────────────────────┤
│ Sex   │ Multiclass{2} │ CategoricalValue{String, UInt32} │
│ Entry │ Count         │ Int32                            │
│ Cens  │ Count         │ Int32                            │
└───────┴───────────────┴──────────────────────────────────┘

Fixing wrong scientific types in X:

X = coerce(X, :Exit=>Continuous, :Entry=>Continuous, :Cens=>Multiclass);
schema(X)
┌───────┬───────────────┬──────────────────────────────────┐
│ names │ scitypes      │ types                            │
├───────┼───────────────┼──────────────────────────────────┤
│ Sex   │ Multiclass{2} │ CategoricalValue{String, UInt32} │
│ Entry │ Continuous    │ Float64                          │
│ Cens  │ Multiclass{2} │ CategoricalValue{Int32, UInt32}  │
└───────┴───────────────┴──────────────────────────────────┘

Loading a built-in supervised dataset:

table = load_iris();
schema(table)
┌──────────────┬───────────────┬──────────────────────────────────┐
│ names        │ scitypes      │ types                            │
├──────────────┼───────────────┼──────────────────────────────────┤
│ sepal_length │ Continuous    │ Float64                          │
│ sepal_width  │ Continuous    │ Float64                          │
│ petal_length │ Continuous    │ Float64                          │
│ petal_width  │ Continuous    │ Float64                          │
│ target       │ Multiclass{3} │ CategoricalValue{String, UInt32} │
└──────────────┴───────────────┴──────────────────────────────────┘

Loading a built-in data set already split into X and y:

X, y = @load_iris;
selectrows(X, 1:4) # selectrows works whenever `Tables.istable(X)==true`.
(sepal_length = [5.1, 4.9, 4.7, 4.6],
 sepal_width = [3.5, 3.0, 3.2, 3.1],
 petal_length = [1.4, 1.4, 1.3, 1.5],
 petal_width = [0.2, 0.2, 0.2, 0.2],)
y[1:4]
4-element CategoricalArray{String,1,UInt32}:
 "setosa"
 "setosa"
 "setosa"
 "setosa"

Splitting data vertically after row shuffling:

channing_train, channing_test = partition(channing, 0.6, rng=123);

Or, if already horizontally split:

(Xtrain, Xtest), (ytrain, ytest) = partition((X, y), 0.6, multi=true, rng=123)
(((sepal_length = [6.7, 5.7, 7.2, 4.4, 5.6, 6.5, 4.4, 6.1, 5.4, 4.9  …  6.4, 5.5, 5.4, 4.8, 6.5, 4.9, 6.5, 6.7, 5.6, 6.4], sepal_width = [3.3, 2.8, 3.0, 2.9, 2.5, 3.0, 3.0, 2.9, 3.9, 2.5  …  3.1, 2.3, 3.7, 3.1, 3.0, 2.4, 2.8, 3.3, 2.9, 2.8], petal_length = [5.7, 4.1, 5.8, 1.4, 3.9, 5.2, 1.3, 4.7, 1.7, 4.5  …  5.5, 4.0, 1.5, 1.6, 5.5, 3.3, 4.6, 5.7, 3.6, 5.6], petal_width = [2.1, 1.3, 1.6, 0.2, 1.1, 2.0, 0.2, 1.4, 0.4, 1.7  …  1.8, 1.3, 0.2, 0.2, 1.8, 1.0, 1.5, 2.5, 1.3, 2.2]), (sepal_length = [6.0, 5.8, 6.7, 5.1, 5.0, 6.3, 5.7, 6.4, 6.1, 5.0  …  6.4, 6.8, 6.9, 6.1, 6.7, 5.0, 7.6, 6.3, 5.1, 5.0], sepal_width = [2.7, 2.6, 3.0, 3.8, 3.4, 2.8, 2.5, 3.2, 2.8, 3.5  …  2.7, 3.2, 3.1, 2.8, 2.5, 3.5, 3.0, 2.5, 3.8, 3.6], petal_length = [5.1, 4.0, 5.2, 1.9, 1.5, 5.1, 5.0, 4.5, 4.7, 1.6  …  5.3, 5.9, 5.4, 4.0, 5.8, 1.3, 6.6, 5.0, 1.6, 1.4], petal_width = [1.6, 1.2, 2.3, 0.4, 0.2, 1.5, 2.0, 1.5, 1.2, 0.6  …  1.9, 2.3, 2.1, 1.3, 1.8, 0.3, 2.1, 1.9, 0.2, 0.2])), (CategoricalValue{String, UInt32}["virginica", "versicolor", "virginica", "setosa", "versicolor", "virginica", "setosa", "versicolor", "setosa", "virginica"  …  "virginica", "versicolor", "setosa", "setosa", "virginica", "versicolor", "versicolor", "virginica", "versicolor", "virginica"], CategoricalValue{String, UInt32}["versicolor", "versicolor", "virginica", "setosa", "setosa", "virginica", "virginica", "versicolor", "versicolor", "setosa"  …  "virginica", "virginica", "virginica", "versicolor", "virginica", "setosa", "virginica", "virginica", "setosa", "setosa"]))

Reference: Model Search

Searching for a supervised model:

X, y = @load_boston
ms = models(matching(X, y))
70-element Vector{NamedTuple{(:name, :package_name, :is_supervised, :abstract_type, :constructor, :deep_properties, :docstring, :fit_data_scitype, :human_name, :hyperparameter_ranges, :hyperparameter_types, :hyperparameters, :implemented_methods, :inverse_transform_scitype, :is_pure_julia, :is_wrapper, :iteration_parameter, :load_path, :package_license, :package_url, :package_uuid, :predict_scitype, :prediction_type, :reporting_operations, :reports_feature_importances, :supports_class_weights, :supports_online, :supports_training_losses, :supports_weights, :target_in_fit, :transform_scitype, :input_scitype, :target_scitype, :output_scitype)}}:
 (name = ARDRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = AdaBoostRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = BaggingRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = BayesianRidgeRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = CatBoostRegressor, package_name = CatBoost, ... )
 (name = ConstantRegressor, package_name = MLJModels, ... )
 (name = DecisionTreeRegressor, package_name = BetaML, ... )
 (name = DecisionTreeRegressor, package_name = DecisionTree, ... )
 (name = DeterministicConstantRegressor, package_name = MLJModels, ... )
 (name = DummyRegressor, package_name = MLJScikitLearnInterface, ... )
 ⋮
 (name = SGDRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = SRRegressor, package_name = SymbolicRegression, ... )
 (name = SVMLinearRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = SVMNuRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = SVMRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = StableForestRegressor, package_name = SIRUS, ... )
 (name = StableRulesRegressor, package_name = SIRUS, ... )
 (name = TheilSenRegressor, package_name = MLJScikitLearnInterface, ... )
 (name = XGBoostRegressor, package_name = XGBoost, ... )
ms[6]
(name = "ConstantRegressor",
 package_name = "MLJModels",
 is_supervised = true,
 abstract_type = Probabilistic,
 constructor = nothing,
 deep_properties = (),
 docstring = "```\nConstantRegressor\n```\n\nThis \"dummy\" probabilis...",
 fit_data_scitype = Tuple{Table, AbstractVector{Continuous}},
 human_name = "constant regressor",
 hyperparameter_ranges = (nothing,),
 hyperparameter_types = ("Type{D} where D<:Distributions.Sampleable",),
 hyperparameters = (:distribution_type,),
 implemented_methods = [:fitted_params, :predict],
 inverse_transform_scitype = Unknown,
 is_pure_julia = true,
 is_wrapper = false,
 iteration_parameter = nothing,
 load_path = "MLJModels.ConstantRegressor",
 package_license = "MIT",
 package_url = "https://github.com/JuliaAI/MLJModels.jl",
 package_uuid = "d491faf4-2d78-11e9-2867-c94bc002c0b7",
 predict_scitype = AbstractVector{ScientificTypesBase.Density{Continuous}},
 prediction_type = :probabilistic,
 reporting_operations = (),
 reports_feature_importances = false,
 supports_class_weights = false,
 supports_online = false,
 supports_training_losses = false,
 supports_weights = false,
 target_in_fit = true,
 transform_scitype = Unknown,
 input_scitype = Table,
 target_scitype = AbstractVector{Continuous},
 output_scitype = Unknown)
models("Tree")
29-element Vector{NamedTuple{(:name, :package_name, :is_supervised, :abstract_type, :constructor, :deep_properties, :docstring, :fit_data_scitype, :human_name, :hyperparameter_ranges, :hyperparameter_types, :hyperparameters, :implemented_methods, :inverse_transform_scitype, :is_pure_julia, :is_wrapper, :iteration_parameter, :load_path, :package_license, :package_url, :package_uuid, :predict_scitype, :prediction_type, :reporting_operations, :reports_feature_importances, :supports_class_weights, :supports_online, :supports_training_losses, :supports_weights, :target_in_fit, :transform_scitype, :input_scitype, :target_scitype, :output_scitype)}}:
 (name = ABODDetector, package_name = OutlierDetectionNeighbors, ... )
 (name = AdaBoostStumpClassifier, package_name = DecisionTree, ... )
 (name = COFDetector, package_name = OutlierDetectionNeighbors, ... )
 (name = DNNDetector, package_name = OutlierDetectionNeighbors, ... )
 (name = DecisionTreeClassifier, package_name = BetaML, ... )
 (name = DecisionTreeClassifier, package_name = DecisionTree, ... )
 (name = DecisionTreeRegressor, package_name = BetaML, ... )
 (name = DecisionTreeRegressor, package_name = DecisionTree, ... )
 (name = EvoTreeClassifier, package_name = EvoTrees, ... )
 (name = EvoTreeCount, package_name = EvoTrees, ... )
 ⋮
 (name = MultitargetKNNClassifier, package_name = NearestNeighborModels, ... )
 (name = MultitargetKNNRegressor, package_name = NearestNeighborModels, ... )
 (name = OneRuleClassifier, package_name = OneRule, ... )
 (name = RandomForestClassifier, package_name = BetaML, ... )
 (name = RandomForestClassifier, package_name = DecisionTree, ... )
 (name = RandomForestRegressor, package_name = BetaML, ... )
 (name = RandomForestRegressor, package_name = DecisionTree, ... )
 (name = RecursiveFeatureElimination, package_name = FeatureSelection, ... )
 (name = SMOTENC, package_name = Imbalance, ... )

A more refined search:

models() do model
    matching(model, X, y) &&
    model.prediction_type == :deterministic &&
    model.is_pure_julia
end;

Searching for an unsupervised model:

models(matching(X))
63-element Vector{NamedTuple{(:name, :package_name, :is_supervised, :abstract_type, :constructor, :deep_properties, :docstring, :fit_data_scitype, :human_name, :hyperparameter_ranges, :hyperparameter_types, :hyperparameters, :implemented_methods, :inverse_transform_scitype, :is_pure_julia, :is_wrapper, :iteration_parameter, :load_path, :package_license, :package_url, :package_uuid, :predict_scitype, :prediction_type, :reporting_operations, :reports_feature_importances, :supports_class_weights, :supports_online, :supports_training_losses, :supports_weights, :target_in_fit, :transform_scitype, :input_scitype, :target_scitype, :output_scitype)}}:
 (name = ABODDetector, package_name = OutlierDetectionNeighbors, ... )
 (name = ABODDetector, package_name = OutlierDetectionPython, ... )
 (name = AffinityPropagation, package_name = MLJScikitLearnInterface, ... )
 (name = AgglomerativeClustering, package_name = MLJScikitLearnInterface, ... )
 (name = AutoEncoder, package_name = BetaML, ... )
 (name = Birch, package_name = MLJScikitLearnInterface, ... )
 (name = BisectingKMeans, package_name = MLJScikitLearnInterface, ... )
 (name = CBLOFDetector, package_name = OutlierDetectionPython, ... )
 (name = CDDetector, package_name = OutlierDetectionPython, ... )
 (name = COFDetector, package_name = OutlierDetectionNeighbors, ... )
 ⋮
 (name = RODDetector, package_name = OutlierDetectionPython, ... )
 (name = RandomForestImputer, package_name = BetaML, ... )
 (name = SODDetector, package_name = OutlierDetectionPython, ... )
 (name = SOSDetector, package_name = OutlierDetectionPython, ... )
 (name = SelfOrganizingMap, package_name = SelfOrganizingMaps, ... )
 (name = SimpleImputer, package_name = BetaML, ... )
 (name = SpectralClustering, package_name = MLJScikitLearnInterface, ... )
 (name = Standardizer, package_name = MLJModels, ... )
 (name = TSVDTransformer, package_name = TSVD, ... )

Getting the metadata entry for a given model type:

info("PCA")
info("RidgeRegressor", pkg="MultivariateStats") # a model type in multiple packages
(name = "RidgeRegressor",
 package_name = "MultivariateStats",
 is_supervised = true,
 abstract_type = Deterministic,
 constructor = nothing,
 deep_properties = (),
 docstring = "```\nRidgeRegressor\n```\n\nA model type for construct...",
 fit_data_scitype =
     Tuple{Table{<:AbstractVector{<:Continuous}}, AbstractVector{Continuous}},
 human_name = "ridge regressor",
 hyperparameter_ranges = (nothing, nothing),
 hyperparameter_types = ("Union{Real, AbstractVecOrMat}", "Bool"),
 hyperparameters = (:lambda, :bias),
 implemented_methods = [:clean!, :fit, :fitted_params, :predict],
 inverse_transform_scitype = Unknown,
 is_pure_julia = true,
 is_wrapper = false,
 iteration_parameter = nothing,
 load_path = "MLJMultivariateStatsInterface.RidgeRegressor",
 package_license = "MIT",
 package_url = "https://github.com/JuliaStats/MultivariateStats.jl",
 package_uuid = "6f286f6a-111f-5878-ab1e-185364afe411",
 predict_scitype = AbstractVector{Continuous},
 prediction_type = :deterministic,
 reporting_operations = (),
 reports_feature_importances = false,
 supports_class_weights = false,
 supports_online = false,
 supports_training_losses = false,
 supports_weights = false,
 target_in_fit = true,
 transform_scitype = Unknown,
 input_scitype = Table{<:AbstractVector{<:Continuous}},
 target_scitype = AbstractVector{Continuous},
 output_scitype = Unknown)

Extracting the model document string (output omitted):

doc("DecisionTreeClassifier", pkg="DecisionTree")

Instantiating a model

Reference: Getting Started, Loading Model Code

Assumes MLJDecisionTreeClassifier is in your environment. Otherwise, try interactive loading with @iload:

Tree = @load DecisionTreeClassifier pkg=DecisionTree
tree = Tree(min_samples_split=5, max_depth=4)
DecisionTreeClassifier(
  max_depth = 4, 
  min_samples_leaf = 1, 
  min_samples_split = 5, 
  min_purity_increase = 0.0, 
  n_subfeatures = 0, 
  post_prune = false, 
  merge_purity_threshold = 1.0, 
  display_depth = 5, 
  feature_importance = :impurity, 
  rng = Random._GLOBAL_RNG())

or

tree = (@load DecisionTreeClassifier)()
tree.min_samples_split = 5
tree.max_depth = 4

Evaluating a model

Reference: Evaluating Model Performance

X, y = @load_boston  # a table and a vector
KNN = @load KNNRegressor
knn = KNN()
evaluate(knn, X, y,
         resampling=CV(nfolds=5),
         measure=[RootMeanSquaredError(), LPLoss(1)])
PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌───┬────────────────────────┬───────────┬─────────────┐
│   │ measure                │ operation │ measurement │
├───┼────────────────────────┼───────────┼─────────────┤
│ A │ RootMeanSquaredError() │ predict   │ 8.77        │
│ B │ LPLoss(                │ predict   │ 6.02        │
│   │   p = 1)               │           │             │
└───┴────────────────────────┴───────────┴─────────────┘
┌───┬───────────────────────────────┬─────────┐
│   │ per_fold                      │ 1.96*SE │
├───┼───────────────────────────────┼─────────┤
│ A │ [8.53, 8.8, 10.7, 9.43, 5.59] │ 1.84    │
│ B │ [6.52, 5.7, 7.65, 6.09, 4.11] │ 1.26    │
└───┴───────────────────────────────┴─────────┘

Note RootMeanSquaredError() has alias rms and LPLoss(1) has aliases l1, mae.

Do measures() to list all losses and scores and their aliases, or refer to the StatisticalMeasures.jl docs.

Basic fit/evaluate/predict by hand

Reference: Getting Started, Machines, Evaluating Model Performance, Performance Measures

crabs = load_crabs() |> DataFrames.DataFrame
schema(crabs)
┌───────┬───────────────┬──────────────────────────────────┐
│ names │ scitypes      │ types                            │
├───────┼───────────────┼──────────────────────────────────┤
│ sp    │ Multiclass{2} │ CategoricalValue{String, UInt32} │
│ sex   │ Multiclass{2} │ CategoricalValue{String, UInt32} │
│ index │ Count         │ Int64                            │
│ FL    │ Continuous    │ Float64                          │
│ RW    │ Continuous    │ Float64                          │
│ CL    │ Continuous    │ Float64                          │
│ CW    │ Continuous    │ Float64                          │
│ BD    │ Continuous    │ Float64                          │
└───────┴───────────────┴──────────────────────────────────┘
y, X = unpack(crabs, ==(:sp), !in([:index, :sex]); rng=123)

Tree = @load DecisionTreeClassifier pkg=DecisionTree
DecisionTreeClassifier(
  max_depth = 2, 
  min_samples_leaf = 1, 
  min_samples_split = 2, 
  min_purity_increase = 0.0, 
  n_subfeatures = 0, 
  post_prune = false, 
  merge_purity_threshold = 1.0, 
  display_depth = 5, 
  feature_importance = :impurity, 
  rng = Random._GLOBAL_RNG())

Bind the model and data together in a machine, which will additionally, store the learned parameters (fitresults) when fit:

mach = machine(tree, X, y)
untrained Machine; caches model-specific representations of data
  model: DecisionTreeClassifier(max_depth = 2, …)
  args: 
    1:	Source @588 ⏎ Table{AbstractVector{Continuous}}
    2:	Source @168 ⏎ AbstractVector{Multiclass{2}}

Split row indices into training and evaluation rows:

train, test = partition(eachindex(y), 0.7); # 70:30 split
([1, 2, 3, 4, 5, 6, 7, 8, 9, 10  …  131, 132, 133, 134, 135, 136, 137, 138, 139, 140], [141, 142, 143, 144, 145, 146, 147, 148, 149, 150  …  191, 192, 193, 194, 195, 196, 197, 198, 199, 200])

Fit on the train data set and evaluate on the test data set:

fit!(mach, rows=train)
yhat = predict(mach, X[test,:])
LogLoss(tol=1e-4)(yhat, y[test])
1.0788055664326648

Note LogLoss() has aliases log_loss and cross_entropy.

Predict on the new data set:

Xnew = (FL = rand(3), RW = rand(3), CL = rand(3), CW = rand(3), BD = rand(3))
predict(mach, Xnew)      # a vector of distributions
3-element UnivariateFiniteVector{Multiclass{2}, String, UInt32, Float64}:
 UnivariateFinite{Multiclass{2}}(B=>0.667, O=>0.333)
 UnivariateFinite{Multiclass{2}}(B=>0.667, O=>0.333)
 UnivariateFinite{Multiclass{2}}(B=>0.667, O=>0.333)
predict_mode(mach, Xnew) # a vector of point-predictions
3-element CategoricalArray{String,1,UInt32}:
 "B"
 "B"
 "B"

More performance evaluation examples

Evaluating model + data directly:

evaluate(tree, X, y,
         resampling=Holdout(fraction_train=0.7, shuffle=true, rng=1234),
         measure=[LogLoss(), Accuracy()])
PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌───┬──────────────────────┬──────────────┬─────────────┐
│   │ measure              │ operation    │ measurement │
├───┼──────────────────────┼──────────────┼─────────────┤
│ A │ LogLoss(             │ predict      │ 1.12        │
│   │   tol = 2.22045e-16) │              │             │
│ B │ Accuracy()           │ predict_mode │ 0.683       │
└───┴──────────────────────┴──────────────┴─────────────┘

If a machine is already defined, as above:

evaluate!(mach,
          resampling=Holdout(fraction_train=0.7, shuffle=true, rng=1234),
          measure=[LogLoss(), Accuracy()])
PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌───┬──────────────────────┬──────────────┬─────────────┐
│   │ measure              │ operation    │ measurement │
├───┼──────────────────────┼──────────────┼─────────────┤
│ A │ LogLoss(             │ predict      │ 1.12        │
│   │   tol = 2.22045e-16) │              │             │
│ B │ Accuracy()           │ predict_mode │ 0.683       │
└───┴──────────────────────┴──────────────┴─────────────┘

Using cross-validation:

evaluate!(mach, resampling=CV(nfolds=5, shuffle=true, rng=1234),
          measure=[LogLoss(), Accuracy()])
PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌───┬──────────────────────┬──────────────┬─────────────┐
│   │ measure              │ operation    │ measurement │
├───┼──────────────────────┼──────────────┼─────────────┤
│ A │ LogLoss(             │ predict      │ 0.748       │
│   │   tol = 2.22045e-16) │              │             │
│ B │ Accuracy()           │ predict_mode │ 0.7         │
└───┴──────────────────────┴──────────────┴─────────────┘
┌───┬───────────────────────────────────┬─────────┐
│   │ per_fold                          │ 1.96*SE │
├───┼───────────────────────────────────┼─────────┤
│ A │ [0.552, 0.534, 0.44, 0.693, 1.52] │ 0.432   │
│ B │ [0.775, 0.7, 0.8, 0.6, 0.625]     │ 0.0866  │
└───┴───────────────────────────────────┴─────────┘

With user-specified train/test pairs of row indices:

f1, f2, f3 = 1:13, 14:26, 27:36
pairs = [(f1, vcat(f2, f3)), (f2, vcat(f3, f1)), (f3, vcat(f1, f2))];
evaluate!(mach,
          resampling=pairs,
          measure=[LogLoss(), Accuracy()])
PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌───┬──────────────────────┬──────────────┬─────────────┐
│   │ measure              │ operation    │ measurement │
├───┼──────────────────────┼──────────────┼─────────────┤
│ A │ LogLoss(             │ predict      │ 4.81        │
│   │   tol = 2.22045e-16) │              │             │
│ B │ Accuracy()           │ predict_mode │ 0.736       │
└───┴──────────────────────┴──────────────┴─────────────┘
┌───┬───────────────────────┬─────────┐
│   │ per_fold              │ 1.96*SE │
├───┼───────────────────────┼─────────┤
│ A │ [5.1, 6.48, 3.07]     │ 2.38    │
│ B │ [0.696, 0.739, 0.769] │ 0.0513  │
└───┴───────────────────────┴─────────┘

Changing a hyperparameter and re-evaluating:

tree.max_depth = 3
evaluate!(mach,
          resampling=CV(nfolds=5, shuffle=true, rng=1234),
          measure=[LogLoss(), Accuracy()])
PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌───┬──────────────────────┬──────────────┬─────────────┐
│   │ measure              │ operation    │ measurement │
├───┼──────────────────────┼──────────────┼─────────────┤
│ A │ LogLoss(             │ predict      │ 1.19        │
│   │   tol = 2.22045e-16) │              │             │
│ B │ Accuracy()           │ predict_mode │ 0.865       │
└───┴──────────────────────┴──────────────┴─────────────┘
┌───┬────────────────────────────────┬─────────┐
│   │ per_fold                       │ 1.96*SE │
├───┼────────────────────────────────┼─────────┤
│ A │ [1.26, 0.2, 0.199, 2.15, 2.15] │ 0.957   │
│ B │ [0.8, 0.95, 0.975, 0.8, 0.8]   │ 0.0877  │
└───┴────────────────────────────────┴─────────┘

Inspecting training results

Fit an ordinary least square model to some synthetic data:

x1 = rand(100)
x2 = rand(100)

X = (x1=x1, x2=x2)
y = x1 - 2x2 + 0.1*rand(100);

OLS = @load LinearRegressor pkg=GLM
ols = OLS()
mach =  machine(ols, X, y) |> fit!
trained Machine; caches model-specific representations of data
  model: LinearRegressor(fit_intercept = true, …)
  args: 
    1:	Source @391 ⏎ Table{AbstractVector{Continuous}}
    2:	Source @870 ⏎ AbstractVector{Continuous}

Get a named tuple representing the learned parameters, human-readable if appropriate:

fitted_params(mach)
(features = [:x1, :x2],
 coef = [1.0185509597588693, -2.0079260319062326],
 intercept = 0.0428330120986475,)

Get other training-related information:

report(mach)
(stderror = [0.007950074963266256, 0.011895448839094727, 0.010413238610251802],
 dof_residual = 97.0,
 vcov = [6.320369192155296e-5 -6.469174245837067e-5 -4.606109081268564e-5; -6.469174245837067e-5 0.00014150170308352007 -1.1698844746586784e-5; -4.606109081268564e-5 -1.1698844746586784e-5 0.00010843553835403888],
 deviance = 0.08764099233311778,
 coef_table = ─────────────────────────────────────────────────────────────────────────────
                 Coef.  Std. Error        t  Pr(>|t|)   Lower 95%   Upper 95%
─────────────────────────────────────────────────────────────────────────────
(Intercept)   0.042833  0.00795007     5.39    <1e-06   0.0270543   0.0586117
x1            1.01855   0.0118954     85.63    <1e-92   0.994942    1.04216
x2           -2.00793   0.0104132   -192.82    <1e-99  -2.02859    -1.98726
─────────────────────────────────────────────────────────────────────────────,)

Basic fit/transform for unsupervised models

Load data:

X, y = @load_iris  # a table and a vector
train, test = partition(eachindex(y), 0.97, shuffle=true, rng=123)
([125, 100, 130, 9, 70, 148, 39, 64, 6, 107  …  110, 59, 139, 21, 112, 144, 140, 72, 109, 41], [106, 147, 47, 5])

Instantiate and fit the model/machine:

PCA = @load PCA
pca = PCA(maxoutdim=2)
mach = machine(pca, X)
fit!(mach, rows=train)
trained Machine; caches model-specific representations of data
  model: PCA(maxoutdim = 2, …)
  args: 
    1:	Source @521 ⏎ Table{AbstractVector{Continuous}}

Transform selected data bound to the machine:

transform(mach, rows=test);
(x1 = [-3.394282685448322, -1.5219827578765053, 2.53824745518522, 2.7299639893931382],
 x2 = [0.547245022374522, -0.36842368617126425, 0.5199299511335688, 0.3448466122232349],)

Transform new data:

Xnew = (sepal_length=rand(3), sepal_width=rand(3),
        petal_length=rand(3), petal_width=rand(3));
transform(mach, Xnew)
(x1 = [4.819063481959949, 4.299795552521177, 4.47496244458803],
 x2 = [-4.985318004795112, -4.8306940413135395, -5.133804939274677],)

Inverting learned transformations

y = rand(100);
stand = Standardizer()
mach = machine(stand, y)
fit!(mach)
z = transform(mach, y);
@assert inverse_transform(mach, z) ≈ y # true
[ Info: Training machine(Standardizer(features = Symbol[], …), …).

Nested hyperparameter tuning

Reference: Tuning Models

Define a model with nested hyperparameters:

Tree = @load DecisionTreeClassifier pkg=DecisionTree
tree = Tree()
forest = EnsembleModel(model=tree, n=300)
ProbabilisticEnsembleModel(
  model = DecisionTreeClassifier(
        max_depth = -1, 
        min_samples_leaf = 1, 
        min_samples_split = 2, 
        min_purity_increase = 0.0, 
        n_subfeatures = 0, 
        post_prune = false, 
        merge_purity_threshold = 1.0, 
        display_depth = 5, 
        feature_importance = :impurity, 
        rng = Random._GLOBAL_RNG()), 
  atomic_weights = Float64[], 
  bagging_fraction = 0.8, 
  rng = Random._GLOBAL_RNG(), 
  n = 300, 
  acceleration = CPU1{Nothing}(nothing), 
  out_of_bag_measure = Any[])

Define ranges for hyperparameters to be tuned:

r1 = range(forest, :bagging_fraction, lower=0.5, upper=1.0, scale=:log10)
NumericRange(0.5 ≤ bagging_fraction ≤ 1.0; origin=0.75, unit=0.25; on log10 scale)
r2 = range(forest, :(model.n_subfeatures), lower=1, upper=4) # nested
NumericRange(1 ≤ model.n_subfeatures ≤ 4; origin=2.5, unit=1.5)

Wrap the model in a tuning strategy:

tuned_forest = TunedModel(model=forest,
                          tuning=Grid(resolution=12),
                          resampling=CV(nfolds=6),
                          ranges=[r1, r2],
                          measure=BrierLoss())
ProbabilisticTunedModel(
  model = ProbabilisticEnsembleModel(
        model = DecisionTreeClassifier(max_depth = -1, …), 
        atomic_weights = Float64[], 
        bagging_fraction = 0.8, 
        rng = Random._GLOBAL_RNG(), 
        n = 300, 
        acceleration = CPU1{Nothing}(nothing), 
        out_of_bag_measure = Any[]), 
  tuning = Grid(
        goal = nothing, 
        resolution = 12, 
        shuffle = true, 
        rng = Random._GLOBAL_RNG()), 
  resampling = CV(
        nfolds = 6, 
        shuffle = false, 
        rng = Random._GLOBAL_RNG()), 
  measure = BrierLoss(), 
  weights = nothing, 
  class_weights = nothing, 
  operation = nothing, 
  range = NumericRange{T, MLJBase.Bounded, Symbol} where T[NumericRange(0.5 ≤ bagging_fraction ≤ 1.0; origin=0.75, unit=0.25; on log10 scale), NumericRange(1 ≤ model.n_subfeatures ≤ 4; origin=2.5, unit=1.5)], 
  selection_heuristic = MLJTuning.NaiveSelection(nothing), 
  train_best = true, 
  repeats = 1, 
  n = nothing, 
  acceleration = CPU1{Nothing}(nothing), 
  acceleration_resampling = CPU1{Nothing}(nothing), 
  check_measure = true, 
  cache = true, 
  compact_history = true, 
  logger = nothing)

Bound the wrapped model to data:

mach = machine(tuned_forest, X, y)
untrained Machine; does not cache data
  model: ProbabilisticTunedModel(model = ProbabilisticEnsembleModel(model = DecisionTreeClassifier(max_depth = -1, …), …), …)
  args: 
    1:	Source @044 ⏎ Table{AbstractVector{Continuous}}
    2:	Source @064 ⏎ AbstractVector{Multiclass{3}}

Fitting the resultant machine optimizes the hyperparameters specified in range, using the specified tuning and resampling strategies and performance measure (possibly a vector of measures), and retrains on all data bound to the machine:

fit!(mach)
trained Machine; does not cache data
  model: ProbabilisticTunedModel(model = ProbabilisticEnsembleModel(model = DecisionTreeClassifier(max_depth = -1, …), …), …)
  args: 
    1:	Source @044 ⏎ Table{AbstractVector{Continuous}}
    2:	Source @064 ⏎ AbstractVector{Multiclass{3}}

Inspecting the optimal model:

F = fitted_params(mach)
(best_model = ProbabilisticEnsembleModel(model = DecisionTreeClassifier(max_depth = -1, …), …),
 best_fitted_params = (fitresult = WrappedEnsemble(atom = DecisionTreeClassifier(max_depth = -1, …), …),),)
F.best_model
ProbabilisticEnsembleModel(
  model = DecisionTreeClassifier(
        max_depth = -1, 
        min_samples_leaf = 1, 
        min_samples_split = 2, 
        min_purity_increase = 0.0, 
        n_subfeatures = 3, 
        post_prune = false, 
        merge_purity_threshold = 1.0, 
        display_depth = 5, 
        feature_importance = :impurity, 
        rng = Random._GLOBAL_RNG()), 
  atomic_weights = Float64[], 
  bagging_fraction = 0.5325205447199813, 
  rng = Random._GLOBAL_RNG(), 
  n = 300, 
  acceleration = CPU1{Nothing}(nothing), 
  out_of_bag_measure = Any[])

Inspecting details of tuning procedure:

r = report(mach);
keys(r)
(:best_model, :best_history_entry, :history, :best_report, :plotting)
r.history[[1,end]]
2-element Vector{@NamedTuple{model::MLJEnsembles.ProbabilisticEnsembleModel{MLJDecisionTreeInterface.DecisionTreeClassifier}, measure::Vector{StatisticalMeasuresBase.RobustMeasure{StatisticalMeasuresBase.FussyMeasure{StatisticalMeasuresBase.RobustMeasure{StatisticalMeasures._BrierLossType}, typeof(StatisticalMeasures.l2_check)}}}, measurement::Vector{Float64}, per_fold::Vector{Vector{Float64}}, evaluation::CompactPerformanceEvaluation{MLJEnsembles.ProbabilisticEnsembleModel{MLJDecisionTreeInterface.DecisionTreeClassifier}, Vector{StatisticalMeasuresBase.RobustMeasure{StatisticalMeasuresBase.FussyMeasure{StatisticalMeasuresBase.RobustMeasure{StatisticalMeasures._BrierLossType}, typeof(StatisticalMeasures.l2_check)}}}, Vector{Float64}, Vector{typeof(predict)}, Vector{Vector{Float64}}, Vector{Vector{Vector{Float64}}}, CV}}}:
 (model = ProbabilisticEnsembleModel(model = DecisionTreeClassifier(max_depth = -1, …), …), measure = [BrierLoss()], measurement = [0.12282044444444412], per_fold = [[-0.0, -0.0, 0.16970577777777746, 0.15881422222222163, 0.15364622222222177, 0.2547564444444439]], evaluation = CompactPerformanceEvaluation(0.123,))
 (model = ProbabilisticEnsembleModel(model = DecisionTreeClassifier(max_depth = -1, …), …), measure = [BrierLoss()], measurement = [0.09621644444444433], per_fold = [[-0.0, 4.444444444473561e-6, 0.12098044444444447, 0.14713688888888846, 0.12693777777777757, 0.182239111111111]], evaluation = CompactPerformanceEvaluation(0.0962,))

Visualizing these results:

using Plots
plot(mach)

Predicting on new data using the optimized model trained on all data:

predict(mach, Xnew)
3-element UnivariateFiniteVector{Multiclass{3}, String, UInt32, Float64}:
 UnivariateFinite{Multiclass{3}}(setosa=>1.0, versicolor=>0.0, virginica=>0.0)
 UnivariateFinite{Multiclass{3}}(setosa=>1.0, versicolor=>0.0, virginica=>0.0)
 UnivariateFinite{Multiclass{3}}(setosa=>1.0, versicolor=>0.0, virginica=>0.0)

Constructing linear pipelines

Reference: Linear Pipelines

Constructing a linear (unbranching) pipeline with a learned target transformation/inverse transformation:

X, y = @load_reduced_ames
KNN = @load KNNRegressor
knn_with_target = TransformedTargetModel(model=KNN(K=3), transformer=Standardizer())
TransformedTargetModelDeterministic(
  model = KNNRegressor(
        K = 3, 
        algorithm = :kdtree, 
        metric = Distances.Euclidean(0.0), 
        leafsize = 10, 
        reorder = true, 
        weights = NearestNeighborModels.Uniform()), 
  transformer = Standardizer(
        features = Symbol[], 
        ignore = false, 
        ordered_factor = false, 
        count = false), 
  inverse = nothing, 
  cache = true)
pipe = (X -> coerce(X, :age=>Continuous)) |> OneHotEncoder() |> knn_with_target
DeterministicPipeline(
  f = Main.var"#15#16"(), 
  one_hot_encoder = OneHotEncoder(
        features = Symbol[], 
        drop_last = false, 
        ordered_factor = true, 
        ignore = false), 
  transformed_target_model_deterministic = TransformedTargetModelDeterministic(
        model = KNNRegressor(K = 3, …), 
        transformer = Standardizer(features = Symbol[], …), 
        inverse = nothing, 
        cache = true), 
  cache = true)

Evaluating the pipeline (just as you would any other model):

pipe.one_hot_encoder.drop_last = true # mutate a nested hyper-parameter
evaluate(pipe, X, y, resampling=Holdout(), measure=RootMeanSquaredError(), verbosity=2)
PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌────────────────────────┬───────────┬─────────────┐
│ measure                │ operation │ measurement │
├────────────────────────┼───────────┼─────────────┤
│ RootMeanSquaredError() │ predict   │ 51200.0     │
└────────────────────────┴───────────┴─────────────┘

Inspecting the learned parameters in a pipeline:

mach = machine(pipe, X, y) |> fit!
F = fitted_params(mach)
F.transformed_target_model_deterministic.model
(tree = NearestNeighbors.KDTree{StaticArraysCore.SVector{56, Float64}, Distances.Euclidean, Float64, StaticArraysCore.SVector{56, Float64}}
  Number of points: 1456
  Dimensions: 56
  Metric: Distances.Euclidean(0.0)
  Reordered: true,)

Constructing a linear (unbranching) pipeline with a static (unlearned) target transformation/inverse transformation:

Tree = @load DecisionTreeRegressor pkg=DecisionTree verbosity=0
tree_with_target = TransformedTargetModel(model=Tree(),
                                          transformer=y -> log.(y),
                                          inverse = z -> exp.(z))
pipe2 = (X -> coerce(X, :age=>Continuous)) |> OneHotEncoder() |> tree_with_target

Creating a homogeneous ensemble of models

Reference: Homogeneous Ensembles

X, y = @load_iris
Tree = @load DecisionTreeClassifier pkg=DecisionTree
tree = Tree()
forest = EnsembleModel(model=tree, bagging_fraction=0.8, n=300)
mach = machine(forest, X, y)
evaluate!(mach, measure=LogLoss())
PerformanceEvaluation object with these fields:
  model, measure, operation,
  measurement, per_fold, per_observation,
  fitted_params_per_fold, report_per_fold,
  train_test_rows, resampling, repeats
Extract:
┌──────────────────────┬───────────┬─────────────┐
│ measure              │ operation │ measurement │
├──────────────────────┼───────────┼─────────────┤
│ LogLoss(             │ predict   │ 0.623       │
│   tol = 2.22045e-16) │           │             │
└──────────────────────┴───────────┴─────────────┘
┌───────────────────────────────────────────────┬─────────┐
│ per_fold                                      │ 1.96*SE │
├───────────────────────────────────────────────┼─────────┤
│ [3.89e-15, 3.89e-15, 0.293, 1.6, 1.55, 0.293] │ 0.657   │
└───────────────────────────────────────────────┴─────────┘

Performance curves

Generate a plot of performance, as a function of some hyperparameter (building on the preceding example)

Single performance curve:

r = range(forest, :n, lower=1, upper=1000, scale=:log10)
curve = learning_curve(mach,
                       range=r,
                       resampling=Holdout(),
                       resolution=50,
                       measure=LogLoss(),
                       verbosity=0)
(parameter_name = "n",
 parameter_scale = :log10,
 parameter_values = [1, 2, 3, 4, 5, 6, 7, 8, 10, 11  …  281, 324, 373, 429, 494, 569, 655, 754, 869, 1000],
 measurements = [9.611640903764574, 4.898240075956947, 4.984567899163308, 4.971179761126414, 4.969096703502204, 4.924535918221357, 4.1861383502179965, 4.193775707005525, 4.221059340527931, 4.240306390820523  …  1.2447361213019805, 1.2518519665458379, 1.2451146571257012, 1.2530584621506828, 1.2577179391626758, 1.2593404412777596, 1.2575466997579048, 1.2505002301869643, 1.2589736092962684, 1.251659059923358],)
using Plots
plot(curve.parameter_values, curve.measurements,
     xlab=curve.parameter_name, xscale=curve.parameter_scale)

Multiple curves:

curve = learning_curve(mach,
                       range=r,
                       resampling=Holdout(),
                       measure=LogLoss(),
                       resolution=50,
                       rng_name=:rng,
                       rngs=4,
                       verbosity=0)
(parameter_name = "n",
 parameter_scale = :log10,
 parameter_values = [1, 2, 3, 4, 5, 6, 7, 8, 10, 11  …  281, 324, 373, 429, 494, 569, 655, 754, 869, 1000],
 measurements = [4.004850376568572 9.611640903764574 15.218431430960575 9.611640903764574; 4.004850376568572 9.611640903764574 15.218431430960575 8.040507294495367; … ; 1.2015280692570036 1.2432561936846804 1.263074474335037 1.276065252402837; 1.206217747506149 1.2397397043540395 1.2646851632092617 1.273676386641689],)
plot(curve.parameter_values, curve.measurements,
     xlab=curve.parameter_name, xscale=curve.parameter_scale)