Resampling

MLJBase.CV — Type

cv = CV(; nfolds=6,  shuffle=nothing, rng=nothing)

Cross-validation resampling strategy, for use in evaluate!, evaluate and tuning.

train_test_pairs(cv, rows)

Returns an nfolds-length iterator of (train, test) pairs of vectors (row indices), where each train and test is a sub-vector of rows. The test vectors are mutually exclusive and exhaust rows. Each train vector is the complement of the corresponding test vector. With no row pre-shuffling, the order of rows is preserved, in the sense that rows coincides precisely with the concatenation of the test vectors, in the order they are generated. The first r test vectors have length n + 1, where n, r = divrem(length(rows), nfolds), and the remaining test vectors have length n.

Pre-shuffling of rows is controlled by rng and shuffle. If rng is an integer, then the CV keyword constructor resets it to MersenneTwister(rng). Otherwise some AbstractRNG object is expected.

If rng is left unspecified, rng is reset to Random.GLOBAL_RNG, in which case rows are only pre-shuffled if shuffle=true is explicitly specified.

source

MLJBase.Holdout — Type

holdout = Holdout(; fraction_train=0.7,
                     shuffle=nothing,
                     rng=nothing)

Holdout resampling strategy, for use in evaluate!, evaluate and in tuning.

train_test_pairs(holdout, rows)

Returns the pair [(train, test)], where train and test are vectors such that rows=vcat(train, test) and length(train)/length(rows) is approximatey equal to fraction_train`.

Pre-shuffling of rows is controlled by rng and shuffle. If rng is an integer, then the Holdout keyword constructor resets it to MersenneTwister(rng). Otherwise some AbstractRNG object is expected.

If rng is left unspecified, rng is reset to Random.GLOBAL_RNG, in which case rows are only pre-shuffled if shuffle=true is specified.

source

MLJBase.Resampler — Type

resampler = Resampler(model=ConstantRegressor(),
                      resampling=CV(),
                      measure=nothing,
                      weights=nothing,
                      operation=predict,
                      repeats = 1,
                      acceleration=default_resource(),
                      check_measure=true)

Resampling model wrapper, used internally by the fit method of TunedModel instances. See `evaluate! for options. Not intended for general use.

Given a machine mach = machine(resampler, args...) one obtains a performance evaluation of the specified model, performed according to the prescribed resampling strategy and other parameters, using data args..., by calling fit!(mach) followed by evaluate(mach). The advantage over using evaluate(model, X, y) is that the latter call always calls fit on the model but fit!(mach) only calls update after the first call.

The sample weights are passed to the specified performance measures that support weights for evaluation.

Important: If weights are left unspecified, then any weight vector w used in constructing the resampler machine, as in resampler_machine = machine(resampler, X, y, w) (which is then used in training the model) will also be used in evaluation.

source

MLJBase.StratifiedCV — Type

stratified_cv = StratifiedCV(; nfolds=6,
                               shuffle=false,
                               rng=Random.GLOBAL_RNG)

Stratified cross-validation resampling strategy, for use in evaluate!, evaluate and in tuning. Applies only to classification problems (OrderedFactor or Multiclass targets).

train_test_pairs(stratified_cv, rows, y)

Returns an nfolds-length iterator of (train, test) pairs of vectors (row indices) where each train and test is a sub-vector of rows. The test vectors are mutually exclusive and exhaust rows. Each train vector is the complement of the corresponding test vector.

Unlike regular cross-validation, the distribution of the levels of the target y corresponding to each train and test is constrained, as far as possible, to replicate that of y[rows] as a whole.

Specifically, the data is split into a number of groups on which y is constant, and each individual group is resampled according to the ordinary cross-validation strategy CV(nfolds=nfolds). To obtain the final (train, test) pairs of row indices, the per-group pairs are collated in such a way that each collated train and test respects the original order of rows (after shuffling, if shuffle=true).

Pre-shuffling of rows is controlled by rng and shuffle. If rng is an integer, then the StratifedCV keyword constructor resets it to MersenneTwister(rng). Otherwise some AbstractRNG object is expected.

If rng is left unspecified, rng is reset to Random.GLOBAL_RNG, in which case rows are only pre-shuffled if shuffle=true is explicitly specified.

source

MLJBase.evaluate! — Method

evaluate!(mach,
          resampling=CV(),
          measure=nothing,
          weights=nothing,
          operation=predict,
          repeats = 1,
          acceleration=default_resource(),
          force=false,
          verbosity=1,
          check_measure=true)

Estimate the performance of a machine mach wrapping a supervised model in data, using the specified resampling strategy (defaulting to 6-fold cross-validation) and measure, which can be a single measure or vector.

Do subtypes(MLJ.ResamplingStrategy) to obtain a list of available resampling strategies. If resampling is not an object of type MLJ.ResamplingStrategy, then a vector of pairs (of the form (train_rows, test_rows) is expected. For example, setting

resampling = [(1:100), (101:200)),
               (101:200), (1:100)]

gives two-fold cross-validation using the first 200 rows of data.

The resampling strategy is applied repeatedly if repeats > 1. For resampling = CV(nfolds=5), for example, this generates a total of 5n test folds for evaluation and subsequent aggregation.

If resampling isa MLJ.ResamplingStrategy then one may optionally restrict the data used in evaluation by specifying rows.

An optional weights vector may be passed for measures that support sample weights (MLJ.supports_weights(measure) == true), which is ignored by those that don't.

Important: If mach already wraps sample weights w (as in mach = machine(model, X, y, w)) then these weights, which are used for training, are automatically passed to the measures for evaluation. However, for evaluation purposes, any weights specified as a keyword argument will take precedence over w.

User-defined measures are supported; see the manual for details.

If no measure is specified, then default_measure(mach.model) is used, unless this default is nothing and an error is thrown.

The acceleration keyword argument is used to specify the compute resource (a subtype of ComputationalResources.AbstractResource) that will be used to accelerate/parallelize the resampling operation.

Although evaluate! is mutating, mach.model and mach.args are untouched.

Return value

A property-accessible object of type PerformanceEvaluation with these properties:

measure: the vector of specified measures
measurements: the corresponding measurements, aggregated across the test folds using the aggregation method defined for each measure (do aggregation(measure) to inspect)
per_fold: a vector of vectors of individual test fold evaluations (one vector per measure)
per_observation: a vector of vectors of individual observation evaluations of those measures for which reports_each_observation(measure) is true, which is otherwise reported missing.