Controls

Step(; n=1)

An iteration control, as in, Step(2).

Train for n more iterations. Will never trigger a stop.

Info(f=identity)

An iteration control, as in, Info(my_loss_function).

Log to Info the value of f(m), where m is the object being iterated. If IterativeControl.expose(m) has been overloaded, then log f(expose(m)) instead.

Can be suppressed by setting the global verbosity level sufficiently low.

See also Warn, Error.

Warn(predicate; f="")

An iteration control, as in, Warn(m -> length(m.cache) > 100, f="Memory low").

If predicate(m) is true, then log to Warn the value of f (or f(IterationControl.expose(m)) if f is a function). Here m is the object being iterated.

Can be suppressed by setting the global verbosity level sufficiently low.

See also Info, Error.

Error(predicate; f="", exception=nothing))

An iteration control, as in, Error(m -> isnan(m.bias), f="Bias overflow!").

If predicate(m) is true, then log at the Error level the value of f (or f(IterationControl.expose(m)) if f is a function) and stop iteration at the end of the current control cycle. Here m is the object being iterated.

Specify exception=... to throw an immediate exception, without waiting to the end of the control cycle.

See also Info, Warn.

Callback(f=_->nothing, stop_if_true=false, stop_message=nothing, raw=false)

An iteration control, as in, Callback(m->put!(v, my_loss_function(m)).

Call f(IterationControl.expose(m)), where m is the object being iterated, unless raw=true, in which case call f(m) (guaranteed if expose has not been overloaded.) If stop_if_true is true, then trigger an early stop if the value returned by f is true, logging the stop_message if specified.

When using with MLJIteration.jl, Callback(f) is the same as WithMachineDo(f).

TimeLimit(; t=0.5)

An early stopping criterion for loss-reporting iterative algorithms.

Stopping is triggered after t hours have elapsed since the stopping criterion was initiated.

Any Julia built-in Real type can be used for t. Subtypes of Period may also be used, as in TimeLimit(t=Minute(30)).

Internally, t is rounded to nearest millisecond.

NumberLimit(; n=100)

An early stopping criterion for loss-reporting iterative algorithms.

A stop is triggered by n consecutive loss updates, excluding "training" loss updates.

If wrapped in a stopper::EarlyStopper, this is the number of calls to done!(stopper).

NumberSinceBest(; n=6)

An early stopping criterion for loss-reporting iterative algorithms.

A stop is triggered when the number of calls to the control, since the lowest value of the loss so far, is n.

If using IterationControl.jl, see also the cusstomizable loss-based stopping criterion, WithLossDo and WithTrainingLossesDo, which have anstopiftrue` option.

WithNumberDo(f=identity, stop_if_true=false, stop_message=nothing)

An iteration control, as in, WithNumberDo(n->put!(my_channel, n)).

Call f(n + 1), where n is the number of already completed control cycles (so, n = 1, 2, 3, ..., unless control is wrapped in a IterationControl.skip).

If stop_if_true is true, then trigger an early stop if the value returned by f is true, logging the stop_message if specified.

WithLossDo(f=identity, stop_if_true=false, stop_message=nothing)

An iteration control, as in, WithLossDo(x->put!(my_losses, x)).

Call f(loss), where loss is current loss.

If stop_if_true is true, then trigger an early stop if the value returned by f is true, logging the stop_message if specified.

WithTrainingLossesDo(f=identity, stop_if_true=false, stop_message=nothing)

An iteration control, as in, WithTrainingLossesDo(v->put!(my_losses, last(v)).

Call f(training_losses), where training_losses is the vector of most recent batch of training losses.

If stop_if_true is true, then trigger an early stop if the value returned by f is true, logging the stop_message if specified.

InvalidValue()

An early stopping criterion for loss-reporting iterative algorithms.

Stop if a loss (or training loss) is NaN, Inf or -Inf (or, more precisely, if isnan(loss) or isinf(loss) is true).

If using IterationControl.jl, see also the cusstomizable loss-based stopping criterion, WithLossDo and WithTrainingLossesDo, which have anstopiftrue` option.

Threshold(; value=0.0)

An early stopping criterion for loss-reporting iterative algorithms.

A stop is triggered as soon as the loss drops below value.

If using IterationControl.jl, see also the cusstomizable loss-based stopping criterion, WithLossDo and WithTrainingLossesDo, which have anstopiftrue` option.

GL(; alpha=2.0)

An early stopping criterion for loss-reporting iterative algorithms.

A stop is triggered when the (rescaled) generalization loss exceeds the threshold alpha.

Terminology. Suppose $E_1, E_2, ..., E_t$ are a sequence of losses, for example, out-of-sample estimates of the loss associated with some iterative machine learning algorithm. Then the generalization loss at time t, is given by

$GL_t = 100 (E_t - E_{opt}) \over |E_{opt}|$

where $E_{opt}$ is the minimum value of the sequence.

Reference: Prechelt, Lutz (1998): "Early Stopping- But When?", in Neural Networks: Tricks of the Trade, ed. G. Orr, Springer..

Patience(; n=5)

An early stopping criterion for loss-reporting iterative algorithms.

A stop is triggered by n consecutive increases in the loss.

Denoted "UPs" in Prechelt, Lutz (1998): "Early Stopping- But When?", in Neural Networks: Tricks of the Trade, ed. G. Orr, Springer..

If using IterationControl.jl, see also the cusstomizable loss-based stopping criterion, WithLossDo and WithTrainingLossesDo, which have anstopiftrue` option.

PQ(; alpha=0.75, k=5, tol=eps(Float64))

A stopping criterion for training iterative supervised learners.

A stop is triggered when Prechelt's progress-modified generalization loss exceeds the threshold $PQ_T > alpha$, or if the training progress drops below $P_j ≤ tol$. Here k is the number of training (in-sample) losses used to estimate the training progress.

Context and explanation of terminology

The training progress at time $j$ is defined by

$P_j = 1000 |M - m|/|m|$

where $M$ is the mean of the last k training losses $F_1, F_2, …, F_k$ and $m$ is the minimum value of those losses.

The progress-modified generalization loss at time $t$ is then given by

$PQ_t = GL_t / P_t$

where $GL_t$ is the generalization loss at time $t$; see GL.

PQ will stop when the following are true:

1. At least k training samples have been collected via done!(c::PQ, loss; training = true) or update_training(c::PQ, loss, state)
2. The last update was an out-of-sample update. (done!(::PQ, loss; training=true) is always false)
3. The progress-modified generalization loss exceeds the threshold $PQ_t > alpha$ OR the training progress stalls $P_j ≤ tol$.

Reference: Prechelt, Lutz (1998): "Early Stopping- But When?", in Neural Networks: Tricks of the Trade, ed. G. Orr, Springer..

Warmup(c::StoppingCriterion, n)

Wait for n updates before checking stopping criterion c for the first time.

Data(my_data; stop_when_exhausted=false)

An iteration control, as in, Data(rand(100)).

In each application of this control a new item from the iterable, data, is retrieved (using iterate) and IterationControl.ingest!(m, item) is called. Here m is the object being iteratively trained.

The control becomes passive once the data iterable is done. To trigger a stop after one passive application of the control, set stop_when_exhausted=true.

IterationControl.skip(control, predicate=1)

An iteration control wrapper.

If predicate is an integer, k: Apply control on every k calls to apply the wrapped control, starting with the kth call.

If predicate is a callable: Apply control as usual when predicate(n + 1) is true but otherwise skip. Here n is the number of control cycles applied so far.

While the application of the underlying control gets skipped, the (unchanged) internal state is nevertheless recorded to the corresponding ControlTrace, but this trace does not start recording until the first application of control:

using IterationControlmodel = IterationControl.SquareRooter(4)julia> traces = IC.train!(    m,    Step(1),    IterationControl.skip(WithNumberDo(x->(@info(x); x)), predicate=2),    NumberLimit(5),)[ Info: 2[ Info: 4[ Info: final loss: 0.0[ Info: final training loss: 0.0[ Info: Stop triggered by NumberLimit(5) stopping criterion.3-element Vector{Any}: ControlTrace{<:Step}, ending (new_iterations = 5, new_steps = 5) ControlTrace{<:Skip}, ending (number = 4, called_back = 4) ControlTrace{<:NumberLimit}, ending (number = 5,)julia> traces[2].number4-element Vector{Int64}: 2 2 4 4

IterationControl.louder(control, by=1)

Wrap control to make in more (or less) verbose. The same as control, but as if the global verbosity were increased by the value by.

IterationControl.with_state_do(
    control,
    f=x->(@info "`$(typeof(control))` state: $x; x)",
    raw_state=false,
)

Wrap control to give access to the public form of its internal state (which is also the object accumulated by the corresponding ControlTrace returned by IterationControl.train!). Acts exactly like control except that f is called on current state. If f is not specified, the control type and state are logged to Info at every update. Useful for debugging custom controls.

If raw_state=true, then the raw internal state is accessed instead. Warning: The form of the raw state can change in any new release of IterationControls.jl.

composite(controls...)

Construct an iteration control that applies the specified controls in sequence. Mainly intended for internal use by IterationControls.jl.