Datasets

Load the full version of the well-known Ames Housing task.

Load a well-known public regression dataset with Continuous features.

Load a well-known crab classification dataset with nominal features.

Load a well-known public classification task with nominal features.

Load a reduced version of the well-known Ames Housing task

Load S&P Stock Market dataset, as used in (An Introduction to Statistical Learning with applications in R)https://rdrr.io/cran/ISLR/man/Smarket.html, by Witten et al (2013), Springer-Verlag, New York.

load_dataset(fpath, coercions)

Load one of standard dataset like Boston etc assuming the file is a comma separated file with a header.

X, y = make_blobs(n=100, p=2; kwargs...)

Generate Gaussian blobs for clustering and classification problems.

Return value

By default, a table X with p columns (features) and n rows (observations), together with a corresponding vector of n Multiclass target observations y, indicating blob membership.

Keyword arguments

• shuffle=true: whether to shuffle the resulting points,

• centers=3: either a number of centers or a c x p matrix with c pre-determined centers,

• cluster_std=1.0: the standard deviation(s) of each blob,

• center_box=(-10. => 10.): the limits of the p-dimensional cube within which the cluster centers are drawn if they are not provided,

• eltype=Float64: machine type of points (any subtype of AbstractFloat).

• rng=Random.GLOBAL_RNG: any AbstractRNG object, or integer to seed a MersenneTwister (for reproducibility).

• as_table=true: whether to return the points as a table (true) or a matrix (false). If false the target y has integer element type.

Example

X, y = make_blobs(100, 3; centers=2, cluster_std=[1.0, 3.0])

X, y = make_circles(n=100; kwargs...)

Generate n labeled points close to two concentric circles for classification and clustering models.

Return value

By default, a table X with 2 columns and n rows (observations), together with a corresponding vector of n Multiclass target observations y. The target is either 0 or 1, corresponding to membership to the smaller or larger circle, respectively.

Keyword arguments

• shuffle=true: whether to shuffle the resulting points,

• noise=0: standard deviation of the Gaussian noise added to the data,

• factor=0.8: ratio of the smaller radius over the larger one,

• eltype=Float64: machine type of points (any subtype of AbstractFloat).

• rng=Random.GLOBAL_RNG: any AbstractRNG object, or integer to seed a MersenneTwister (for reproducibility).

• as_table=true: whether to return the points as a table (true) or a matrix (false). If false the target y has integer element type.

Example

X, y = make_circles(100; noise=0.5, factor=0.3)

    make_moons(n::Int=100; kwargs...)

Generates labeled two-dimensional points lying close to two interleaved semi-circles, for use with classification and clustering models.

Return value

By default, a table X with 2 columns and n rows (observations), together with a corresponding vector of n Multiclass target observations y. The target is either 0 or 1, corresponding to membership to the left or right semi-circle.

Keyword arguments

• shuffle=true: whether to shuffle the resulting points,

• noise=0.1: standard deviation of the Gaussian noise added to the data,

• xshift=1.0: horizontal translation of the second center with respect to the first one.

• yshift=0.3: vertical translation of the second center with respect to the first one.

• eltype=Float64: machine type of points (any subtype of AbstractFloat).

• rng=Random.GLOBAL_RNG: any AbstractRNG object, or integer to seed a MersenneTwister (for reproducibility).

• as_table=true: whether to return the points as a table (true) or a matrix (false). If false the target y has integer element type.

Example

X, y = make_moons(100; noise=0.5)

make_regression(n, p; kwargs...)

Generate Gaussian input features and a linear response with Gaussian noise, for use with regression models.

Return value

By default, a table X with p columns and n rows (observations), together with a corresponding vector of n Continuous target observations y.

Keywords

• `intercept=true: whether to generate data from a model with intercept,

• sparse=0: portion of the generating weight vector that is sparse,

• noise=0.1: standard deviation of the Gaussian noise added to the response,

• outliers=0: portion of the response vector to make as outliers by adding a random quantity with high variance. (Only applied if binary is false)

• binary=false: whether the target should be binarized (via a sigmoid).

• eltype=Float64: machine type of points (any subtype of AbstractFloat).

• rng=Random.GLOBAL_RNG: any AbstractRNG object, or integer to seed a MersenneTwister (for reproducibility).

• as_table=true: whether to return the points as a table (true) or a matrix (false).

Example

X, y = make_regression(100, 5; noise=0.5, sparse=0.2, outliers=0.1)

augment_X(X, fit_intercept)

Given a matrix X, append a column of ones if fit_intercept is true. See make_regression.

finalize_Xy(X, y, shuffle, as_table, eltype, rng; clf)

Internal function to finalize the make_* functions.

Add outliers to portion s of vector.

runif_ab(rng, n, p, a, b)

Internal function to generate n points in [a, b]ᵖ uniformly at random.

sigmoid(x)

Return the sigmoid computed in a numerically stable way:

$σ(x) = 1/(1+exp(-x))$

sparsify!(rng, θ, s)

Make portion s of vector θ exactly 0.

complement(folds, i)

The complement of the ith fold of folds in the concatenation of all elements of folds. Here folds is a vector or tuple of integer vectors, typically representing row indices or a vector, matrix or table.

complement(([1,2], [3,], [4, 5]), 2) # [1 ,2, 4, 5]

corestrict(X, folds, i)

The restriction of X, a vector, matrix or table, to the complement of the ith fold of folds, where folds is a tuple of vectors of row indices.

The method is curried, so that corestrict(folds, i) is the operator on data defined by corestrict(folds, i)(X) = corestrict(X, folds, i).

Example

folds = ([1, 2], [3, 4, 5],  [6,])
corestrict([:x1, :x2, :x3, :x4, :x5, :x6], folds, 2) # [:x1, :x2, :x6]

partition(X, fractions...;
          shuffle=nothing, rng=Random.GLOBAL_RNG, stratify=nothing)

Splits the vector or matrix X into a tuple of vectors or matrices whose numbers of rows are given by the corresponding fractions of length(nrows(X)), where valid fractions are in (0,1) and sum up to less than

1. The last fraction is not provided, as it is inferred from the preceding ones.

X can also be any object which implements the Tables.jl interface according to Tables.istable.

So, for example,

julia> partition(1:1000, 0.8)
([1,...,800], [801,...,1000])

julia> partition(1:1000, 0.2, 0.7)
([1,...,200], [201,...,900], [901,...,1000])

julia> partition(reshape(1:10, 5, 2), 0.2, 0.4)
([1 6], [2 7; 3 8], [4 9; 5 10])

Keywords

• shuffle=nothing: if set to true, shuffles the rows before taking fractions.
• rng=Random.GLOBAL_RNG: specifies the random number generator to be used, can be an integer seed. If specified, and shuffle === nothing is interpreted as true.
• stratify=nothing: if a vector is specified, the partition will match the stratification of the given vector. In that case, shuffle cannot be false.

restrict(X, folds, i)

The restriction of X, a vector, matrix or table, to the ith fold of folds, where folds is a tuple of vectors of row indices.

The method is curried, so that restrict(folds, i) is the operator on data defined by restrict(folds, i)(X) = restrict(X, folds, i).

Example

folds = ([1, 2], [3, 4, 5],  [6,])
restrict([:x1, :x2, :x3, :x4, :x5, :x6], folds, 2) # [:x3, :x4, :x5]

See also corestrict

t1, t2, ...., tk = unnpack(table, f1, f2, ... fk;
                           wrap_singles=false,
                           shuffle=false,
                           rng::Union{AbstractRNG,Int,Nothing}=nothing)

Split any Tables.jl compatible table into smaller tables (or vectors) t1, t2, ..., tk by making selections without replacement from the column names defined by the filters f1, f2, ..., fk. A filter is any object f such that f(name) is true or false for each column name::Symbol of table.

Whenever a returned table contains a single column, it is converted to a vector unless wrap_singles=true.

Scientific type conversions can be optionally specified (note semicolon):

unpack(table, t...; wrap_singles=false, col1=>scitype1, col2=>scitype2, ... )

If shuffle=true then the rows of table are first shuffled, using the global RNG, unless rng is specified; if rng is an integer, it specifies the seed of an automatically generated Mersenne twister. If rng is specified then shuffle=true is implicit.

Example

julia> table = DataFrame(x=[1,2], y=['a', 'b'], z=[10.0, 20.0], w=["A", "B"])
julia> Z, XY = unpack(table, ==(:z), !=(:w);
               :x=>Continuous, :y=>Multiclass)
julia> XY
2×2 DataFrame
│ Row │ x       │ y            │
│     │ Float64 │ Categorical… │
├─────┼─────────┼──────────────┤
│ 1   │ 1.0     │ 'a'          │
│ 2   │ 2.0     │ 'b'          │

julia> Z
2-element Array{Float64,1}:
 10.0
 20.0

transform(e::Union{CategoricalElement,CategoricalArray,CategoricalPool},  X)

Transform the specified object X into a categorical version, using the pool contained in e. Here X is a raw value (an element of levels(e)) or an AbstractArray of such values.

```julia v = categorical(["x", "y", "y", "x", "x"]) julia> transform(v, "x") CategoricalValue{String,UInt32} "x"

julia> transform(v[1], ["x" "x"; missing "y"]) 2×2 CategoricalArray{Union{Missing, Symbol},2,UInt32}: "x" "x" missing "y"