TunedModel

tuned_model = TunedModel(; model=<model to be mutated>,
                         tuning=RandomSearch(),
                         resampling=Holdout(),
                         range=nothing,
                         measure=nothing,
                         n=default_n(tuning, range),
                         operation=nothing,
                         other_options...)

Construct a model wrapper for hyper-parameter optimization of a supervised learner, specifying the tuning strategy and model whose hyper-parameters are to be mutated.

tuned_model = TunedModel(; models=<models to be compared>,
                         resampling=Holdout(),
                         measure=nothing,
                         n=length(models),
                         operation=nothing,
                         other_options...)

Construct a wrapper for multiple models, for selection of an optimal one (equivalent to specifying tuning=Explicit() and range=models above). Elements of the iterator models need not have a common type, but they must all be Deterministic or all be Probabilistic and this is not checked but inferred from the first element generated.

See below for a complete list of options.

Training

Calling fit!(mach) on a machine mach=machine(tuned_model, X, y) or mach=machine(tuned_model, X, y, w) will:

• Instigate a search, over clones of model, with the hyperparameter mutations specified by range, for a model optimizing the specified measure, using performance evaluations carried out using the specified tuning strategy and resampling strategy. In the case models is explictly listed, the search is instead over the models generated by the iterator models.
• Fit an internal machine, based on the optimal model fitted_params(mach).best_model, wrapping the optimal model object in all the provided data X, y(, w). Calling predict(mach, Xnew) then returns predictions on Xnew of this internal machine. The final train can be supressed by setting train_best=false.

Search space

The range objects supported depend on the tuning strategy specified. Query the strategy docstring for details. To optimize over an explicit list v of models of the same type, use strategy=Explicit() and specify model=v[1] and range=v.

The number of models searched is specified by n. If unspecified, then MLJTuning.default_n(tuning, range) is used. When n is increased and fit!(mach) called again, the old search history is re-instated and the search continues where it left off.

Measures (metrics)

If more than one measure is specified, then only the first is optimized (unless strategy is multi-objective) but the performance against every measure specified will be computed and reported in report(mach).best_performance and other relevant attributes of the generated report. Options exist to pass per-observation weights or class weights to measures; see below.

Important. If a custom measure, my_measure is used, and the measure is a score, rather than a loss, be sure to check that MLJ.orientation(my_measure) == :score to ensure maximization of the measure, rather than minimization. Override an incorrect value with MLJ.orientation(::typeof(my_measure)) = :score.

Accessing the fitted parameters and other training (tuning) outcomes

A Plots.jl plot of performance estimates is returned by plot(mach) or heatmap(mach).

Once a tuning machine mach has bee trained as above, then fitted_params(mach) has these keys/values:

The named tuple report(mach) includes these keys/values:

plus other key/value pairs specific to the tuning strategy.

Each element of history is a property-accessible object with these properties:

Complete list of key-word options

• model: Supervised model prototype that is cloned and mutated to generate models for evaluation
• models: Alternatively, an iterator of MLJ models to be explicitly evaluated. These may have varying types.
• tuning=RandomSearch(): tuning strategy to be applied (eg, Grid()). See the Tuning Models section of the MLJ manual for a complete list of options.
• resampling=Holdout(): resampling strategy (eg, Holdout(), CV()), StratifiedCV()) to be applied in performance evaluations
• measure: measure or measures to be applied in performance evaluations; only the first used in optimization (unless the strategy is multi-objective) but all reported to the history
• weights: per-observation weights to be passed the measure(s) in performance evaluations, where supported. Check support with supports_weights(measure).
• class_weights: class weights to be passed the measure(s) in performance evaluations, where supported. Check support with supports_class_weights(measure).
• repeats=1: for generating train/test sets multiple times in resampling ("Monte Carlo" resampling); see evaluate! for details
• operation/operations - One of predict, predict_mean, predict_mode, predict_median, or predict_joint, or a vector of these of the same length as measure/measures. Automatically inferred if left unspecified.
• range: range object; tuning strategy documentation describes supported types
• selection_heuristic: the rule determining how the best model is decided. According to the default heuristic, NaiveSelection(), measure (or the first element of measure) is evaluated for each resample and these per-fold measurements are aggregrated. The model with the lowest (resp. highest) aggregate is chosen if the measure is a :loss (resp. a :score).
• n: number of iterations (ie, models to be evaluated); set by tuning strategy if left unspecified
• train_best=true: whether to train the optimal model
• acceleration=default_resource(): mode of parallelization for tuning strategies that support this
• acceleration_resampling=CPU1(): mode of parallelization for resampling
• check_measure=true: whether to check measure is compatible with the specified model and operation)
• cache=true: whether to cache model-specific representations of user-suplied data; set to false to conserve memory. Speed gains likely limited to the case resampling isa Holdout.
• compact_history=true: whether to write CompactPerformanceEvaluation](@ref) or regular PerformanceEvaluation objects to the history (accessed via the :evaluation key); the compact form excludes some fields to conserve memory.
• logger=default_logger(): a logger for externally reporting model performance evaluations, such as an MLJFlow.Logger instance. On startup, default_logger()=nothing; use default_logger(logger) to set a global logger.