MLFlow

Base type which defines location and version for MLFlow API service.

Fields

• apiroot::String: API root URL, e.g. http://localhost:5000/api
• apiversion::Union{Integer, AbstractFloat}: used API version, e.g. 2.0
• headers::Dict: HTTP headers to be provided with the REST API requests (useful for authetication tokens)

Default is false, using the REST API endpoint.

Constructors

• MLFlow(apiroot; apiversion=2.0,headers=Dict())
• MLFlow() - defaults to MLFlow(ENV["MLFLOW_TRACKING_URI"]) or MLFlow("http://localhost:5000/api")

Examples

mlf = MLFlow()

remote_url="https://<your-server>.cloud.databricks.com"; # address of your remote server
mlf = MLFlow(remote_url, headers=Dict("Authorization" => "Bearer <your-secret-token>"))

MLFlowExperiment

Represents an MLFlow experiment.

Fields

• name::String: experiment name.
• lifecycle_stage::String: life cycle stage, one of ["active", "deleted"]
• experiment_id::Integer: experiment identifier.
• tags::Any: list of tags.
• artifact_location::String: where are experiment artifacts stored.

Constructors

• MLFlowExperiment(name, lifecycle_stage, experiment_id, tags, artifact_location)
• MLFlowExperiment(exp::Dict{String,Any})

MLFlowRun

Represents an MLFlow run.

Fields

• info::MLFlowRunInfo: Run metadata.
• data::MLFlowRunData: Run data.

Constructors

• MLFlowRun(rundata::MLFlowRunData)
• MLFlowRun(runinfo::MLFlowRunInfo)
• MLFlowRun(info::Dict{String,Any})
• MLFlowRun(info::Dict{String,Any}, data::Dict{String,Any})

MLFlowRunInfo

Represents run metadata.

Fields

• run_id::String: run identifier.
• experiment_id::Integer: experiment identifier.
• status::MLFlowRunStatus: run status.
• run_name::String: run name.
• start_time::Union{Int64,Missing}: when was the run started, UNIX time in milliseconds.
• end_time::Union{Int64,Missing}: when did the run end, UNIX time in milliseconds.
• artifact_uri::String: where are artifacts from this run stored.
• lifecycle_stage::String: one of active or deleted.

Constructors

• MLFlowRunInfo(run_id, experiment_id, status, run_name, start_time, end_time, artifact_uri, lifecycle_stage)
• MLFlowRunInfo(info::Dict{String,Any})

MLFlowRunData

Represents run data.

Fields

• metrics::Dict{String,MLFlowRunDataMetric}: run metrics.
• params::Dict{String,MLFlowRunDataParam}: run parameters.
• tags: list of run tags.

Constructors

• MLFlowRunData(data::Dict{String,Any})

MLFlowRunDataParam

Represents a parameter.

Fields

• key::String: parameter identifier.
• value::String: parameter value.

Constructors

• MLFlowRunDataParam(d::Dict{String,String})

MLFlowRunDataMetric

Represents a metric.

Fields

• key::String: metric identifier.
• value::Float64: metric value.
• step::Int64: step.
• timestamp::Int64: timestamp in UNIX time in milliseconds.

Constructors

• MLFlowRunDataMetric(d::Dict{String,Any})

MLFlowRunStatus

Represents the status of an MLFlow Run.

Fields

• status::String: one of RUNNING/SCHEDULED/FINISHED/FAILED/KILLED

Constructors

• MLFlowRunStatus(status::String)

MLFlowArtifactFileInfo

Metadata of a single artifact file – result of listartifacts.

Fields

• filepath::String: File path, including the root artifact directory of a run.
• filesize::Int64: Size in bytes.

MLFlowArtifactDirInfo

Metadata of a single artifact directory – result of listartifacts.

Fields

• dirpath::String: Directory path, including the root artifact directory of a run.

createexperiment(mlf::MLFlow; name=missing, artifact_location=missing, tags=missing)

Creates an MLFlow experiment.

Arguments

• mlf: MLFlow configuration.
• name: experiment name. If not specified, MLFlow sets it.
• artifact_location: directory where artifacts of this experiment will be stored. If not specified, MLFlow uses its default configuration.
• tags: a Dictionary of key-values which tag the experiment.

Returns

An object of type MLFlowExperiment.

getexperiment(mlf::MLFlow, experiment_id::Integer)

Retrieves an MLFlow experiment by id.

Arguments

• mlf: MLFlow configuration.
• experiment_id: Experiment identifier.

Returns

An instance of type MLFlowExperiment

getexperiment(mlf::MLFlow, experiment_name::String)

Retrieves an MLFlow experiment by name.

Arguments

• mlf: MLFlow configuration.
• experiment_name: Experiment name.

Returns

An instance of type MLFlowExperiment

getorcreateexperiment(mlf::MLFlow, experiment_name::String; artifact_location=missing, tags=missing)

Gets an experiment if one alrady exists, or creates a new one.

Arguments

• mlf: MLFlow configuration.
• experiment_name: Experiment name.
• artifact_location: directory where artifacts of this experiment will be stored. If not specified, MLFlow uses its default configuration.
• tags: a Dictionary of key-values which tag the experiment.

Returns

An instance of type MLFlowExperiment

deleteexperiment(mlf::MLFlow, experiment_id::Integer)

Deletes an MLFlow experiment.

Arguments

• mlf: MLFlow configuration.
• experiment_id: experiment identifier.

Returns

true if successful. Otherwise, raises exception.

deleteexperiment(mlf::MLFlow, experiment::MLFlowExperiment)

Deletes an MLFlow experiment.

Arguments

• mlf: MLFlow configuration.
• experiment: an object of type MLFlowExperiment

Dispatches to deleteexperiment(mlf::MLFlow, experiment_id::Integer).

searchexperiments(mlf::MLFlow)

Searches for experiments in an MLFlow instance.

Arguments

• mlf: MLFlow configuration.

Keywords

• filter::String: filter as defined in MLFlow documentation
• filter_attributes::AbstractDict{K,V}: if provided, filter is automatically generated based on filter_attributes using generatefilterfromattributes. One can only provide either filter or filter_attributes, but not both.
• run_view_type::String: one of ACTIVE_ONLY, DELETED_ONLY, or ALL.
• max_results::Integer: 50,000 by default.
• order_by::String: as defined in MLFlow documentation
• page_token::String: paging functionality, handled automatically. Not meant to be passed by the user.

Returns

• vector of MLFlowExperiment experiments that were found in the MLFlow instance

restoreexperiment(mlf::MLFlow, experiment_id::Integer)

Restores a deleted MLFlow experiment.

Arguments

• mlf: MLFlow configuration.
• experiment_id: experiment identifier.

Returns

true if successful. Otherwise, raises exception.

createrun(mlf::MLFlow, experiment_id; run_name=missing, start_time=missing, tags=missing)

Creates a run associated to an experiment.

Arguments

• mlf: MLFlow configuration.
• experiment_id: experiment identifier.

Keywords

• run_name: run name. If not specified, MLFlow sets it.
• start_time: if provided, must be a UNIX timestamp in milliseconds. By default, set to current time.
• tags: if provided, must be a key-value structure such as a dictionary.

Returns

• an instance of type MLFlowRun

createrun(mlf::MLFlow, experiment::MLFlowExperiment; run_name=missing, start_time=missing, tags=missing)

Dispatches to createrun(mlf::MLFlow, experiment_id; run_name=run_name, start_time=start_time, tags=tags)

getrun(mlf::MLFlow, run_id)

Retrieves information about an MLFlow run.

Arguments

• mlf: MLFlow configuration.
• run_id::String: run identifier.

Returns

• an instance of type MLFlowRun

updaterun(mlf::MLFlow, run, status; end_time=missing)

Updates the status of an experiment's run.

Arguments

• mlf: MLFlow configuration.
• run: one of MLFlowRun, MLFlowRunInfo, or String.
• status: either String and one of ["RUNNING", "SCHEDULED", "FINISHED", "FAILED", "KILLED"], or an instance of MLFlowRunStatus

Keywords

• run_name: if provided, must be a String. By default, not set.
• end_time: if provided, must be a UNIX timestamp in milliseconds. By default, set to current time.

deleterun(mlf::MLFlow, run)

Deletes an experiment's run.

Arguments

• mlf: MLFlow configuration.
• run: one of MLFlowRun, MLFlowRunInfo, or String.

Returns

true if successful.

searchruns(mlf::MLFlow, experiment_ids)

Searches for runs in an experiment.

Arguments

• mlf: MLFlow configuration.
• experiment_ids::AbstractVector{Integer}: experiment_ids in which to search for runs. Can also be a single Integer.

Keywords

• filter::String: filter as defined in MLFlow documentation
• filter_params::AbstractDict{K,V}: if provided, filter is automatically generated based on filter_params using generatefilterfromparams. One can only provide either filter or filter_params, but not both.
• run_view_type::String: one of ACTIVE_ONLY, DELETED_ONLY, or ALL.
• max_results::Integer: 50,000 by default.
• order_by::String: as defined in MLFlow documentation
• page_token::String: paging functionality, handled automatically. Not meant to be passed by the user.

Returns

• vector of MLFlowRun runs that were found in the list of experiments.

logparam(mlf::MLFlow, run, key, value)
logparam(mlf::MLFlow, run, kv)

Associates a key/value pair of parameters to the particular run.

Arguments

• mlf: MLFlow configuration.
• run: one of MLFlowRun, MLFlowRunInfo, or String.
• key: parameter key (name). Automatically converted to string before sending to MLFlow because this is the only type that MLFlow supports.
• value: parameter value. Automatically converted to string before sending to MLFlow because this is the only type that MLFlow supports.

One could also specify kv::Dict instead of separate key and value arguments.

logmetric(mlf::MLFlow, run, key, value::T; timestamp, step) where T<:Real
logmetric(mlf::MLFlow, run, key, values::AbstractArray{T}; timestamp, step) where T<:Real

Logs a metric value (or values) against a particular run.

Arguments

• mlf: MLFlow configuration.
• run: one of MLFlowRun, MLFlowRunInfo, or String
• key: metric name.
• value: metric value, must be numeric.

Keywords

• timestamp: if provided, must be a UNIX timestamp in milliseconds. By default, set to current time.
• step: step at which the metric value has been taken.

logbatch(mlf::MLFlow, run_id::String, metrics, params, tags)

Logs a batch of metrics, parameters and tags to an experiment run.

Arguments

• mlf::MLFlow: MLFlow onfiguration.
• run_id::String: ID of the run to log to.
• metrics: a vector of MLFlowRunDataMetric or a vector of

NamedTuples of (name, value, timestamp).

• params: a vector of MLFlowRunDataParam or a vector of NamedTuples

of (name, value).

• tags: a vector of strings.

logartifact(mlf::MLFlow, run, basefilename, data)

Stores an artifact (file) in the run's artifact location.

Arguments

• mlf::MLFlow: MLFlow onfiguration. Currently not used, but when this method is extended to support S3, information from mlf will be needed.
• run: one of MLFlowRun, MLFlowRunInfo or String.
• basefilename: name of the file to be written.
• data: artifact content, an object that can be written directly to a file handle.

Throws

• an ErrorException if an exception occurs during writing artifact.

Returns

path of the artifact that was created.

logartifact(mlf::MLFlow, run, filepath)

Stores an artifact (file) in the run's artifact location. The name of the artifact is calculated using basename(filepath).

Dispatches on logartifact(mlf::MLFlow, run, basefilename, data) where data is the contents of filepath.

Throws

• an ErrorException if filepath does not exist.
• an exception if such occurs while trying to read the contents of filepath.

listartifacts(mlf::MLFlow, run)

Lists the artifacts associated with an experiment run. According to MLFlow documentation, this API endpoint should return paged results, similar to searchruns. However, after some experimentation, this doesn't seem to be the case. Therefore, the paging functionality is not implemented here.

Arguments

• mlf::MLFlow: MLFlow onfiguration. Currently not used, but when this method is extended to support S3, information from mlf will be needed.
• run: one of MLFlowRun, MLFlowRunInfo or String.

Keywords

• path::String: path of a directory within the artifact location. If set, returns the contents of the directory. By default, this is the root directory of the artifacts.
• maxdepth::Int64: depth of listing. Default is 1. This will only return the files/directories in the current path. To return all artifacts files and directories, use maxdepth=-1.

Returns

A vector of Union{MLFlowArtifactFileInfo,MLFlowArtifactDirInfo}.

mlfget(mlf, endpoint; kwargs...)

Performs a HTTP GET to a specified endpoint. kwargs are turned into GET params.

mlfpost(mlf, endpoint; kwargs...)

Performs a HTTP POST to the specified endpoint. kwargs are converted to JSON and become the POST body.

uri(mlf::MLFlow, endpoint="", query=missing)

Retrieves an URI based on mlf, endpoint, and, optionally, query.

Examples

MLFlowClient.uri(mlf, "experiments/get", Dict(:experiment_id=>10))

generatefilterfromentity_type(filter_params::AbstractDict{K,V}, entity_type::String) where {K,V}

Generates a filter string from filter_params dictionary and entity_type.

Arguments

• filter_params: dictionary to use for filter generation.
• entity_type: entity type to use for filter generation.

Returns

A string that can be passed as filter to searchruns.

Examples

generatefilterfromentity_type(Dict("paramkey1" => "paramvalue1", "paramkey2" => "paramvalue2"), "param")

generatefilterfromparams(filter_params::AbstractDict{K,V}) where {K,V}

Generates a filter string from filter_params dictionary and param entity type.

generatefilterfrommattributes(filter_attributes::AbstractDict{K,V}) where {K,V}

Generates a filter string from filter_attributes dictionary and attribute entity type.