OneClassSVM
OneClassSVM
A model type for constructing a one-class support vector machine, based on LIBSVM.jl, and implementing the MLJ model interface.
From MLJ, the type can be imported using
OneClassSVM = @load OneClassSVM pkg=LIBSVM
Do model = OneClassSVM()
to construct an instance with default hyper-parameters. Provide keyword arguments to override hyper-parameter defaults, as in OneClassSVM(kernel=...)
.
Reference for algorithm and core C-library: C.-C. Chang and C.-J. Lin (2011): "LIBSVM: a library for support vector machines." ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27. Updated at https://www.csie.ntu.edu.tw/~cjlin/papers/libsvm.pdf.
This model is an outlier detection model delivering raw scores based on the decision function of a support vector machine. Like the NuSVC
classifier, it uses the nu
re-parameterization of the cost
parameter appearing in standard support vector classification SVC
.
To extract normalized scores ("probabilities") wrap the model using ProbabilisticDetector
from OutlierDetection.jl. For threshold-based classification, wrap the probabilistic model using MLJ's BinaryThresholdPredictor
. Examples of wrapping appear below.
Training data
In MLJ or MLJBase, bind an instance model
to data with:
mach = machine(model, X, y)
where
X
: any table of input features (eg, aDataFrame
) whose columns each haveContinuous
element scitype; check column scitypes withschema(X)
Train the machine using fit!(mach, rows=...)
.
Hyper-parameters
kernel=LIBSVM.Kernel.RadialBasis
: either an object that can be called, as inkernel(x1, x2)
, or one of the built-in kernels from the LIBSVM.jl package listed below. Herex1
andx2
are vectors whose lengths match the number of columns of the training dataX
(see "Examples" below).LIBSVM.Kernel.Linear
:(x1, x2) -> x1'*x2
LIBSVM.Kernel.Polynomial
:(x1, x2) -> gamma*x1'*x2 + coef0)^degree
LIBSVM.Kernel.RadialBasis
:(x1, x2) -> (exp(-gamma*norm(x1 - x2)^2))
LIBSVM.Kernel.Sigmoid
:(x1, x2) - > tanh(gamma*x1'*x2 + coef0)
Here
gamma
,coef0
,degree
are other hyper-parameters. Serialization of models with user-defined kernels comes with some restrictions. See LIVSVM.jl issue91gamma = 0.0
: kernel parameter (see above); ifgamma==-1.0
thengamma = 1/nfeatures
is used in training, wherenfeatures
is the number of features (columns ofX
). Ifgamma==0.0
thengamma = 1/(var(Tables.matrix(X))*nfeatures)
is used. Actual value used appears in the report (see below).coef0 = 0.0
: kernel parameter (see above)degree::Int32 = Int32(3)
: degree in polynomial kernel (see above)nu=0.5
(range (0, 1]): An upper bound on the fraction of margin errors and a lower bound of the fraction of support vectors. Denotedν
in the cited paper. Changingnu
changes the thickness of the margin (a neighborhood of the decision surface) and a margin error is said to have occurred if a training observation lies on the wrong side of the surface or within the margin.cachesize=200.0
cache memory size in MBtolerance=0.001
: tolerance for the stopping criterionshrinking=true
: whether to use shrinking heuristics
Operations
transform(mach, Xnew)
: return scores for outlierness, given featuresXnew
having the same scitype asX
above. The greater the score, the more likely it is an outlier. This score is based on the SVM decision function. For normalized scores, wrapmodel
usingProbabilisticDetector
from OutlierDetection.jl and callpredict
instead, and for threshold-based classification, wrap again usingBinaryThresholdPredictor
. See the examples below.
Fitted parameters
The fields of fitted_params(mach)
are:
libsvm_model
: the trained model object created by the LIBSVM.jl packageorientation
: this equals1
if the decision function forlibsvm_model
is increasing with increasing outlierness, and-1
if it is decreasing instead. Correspondingly, thelibsvm_model
attachestrue
to outliers in the first case, andfalse
in the second. (Thescores
given in the MLJ report and generated byMLJ.transform
already correct for this ambiguity, which is therefore only an issue for users directly accessinglibsvm_model
.)
Report
The fields of report(mach)
are:
gamma
: actual value of the kernel parametergamma
used in training
Examples
Generating raw scores for outlierness
using MLJ
import LIBSVM
import StableRNGs.StableRNG
OneClassSVM = @load OneClassSVM pkg=LIBSVM ## model type
model = OneClassSVM(kernel=LIBSVM.Kernel.Polynomial) ## instance
rng = StableRNG(123)
Xmatrix = randn(rng, 5, 3)
Xmatrix[1, 1] = 100.0
X = MLJ.table(Xmatrix)
mach = machine(model, X) |> fit!
## training scores (outliers have larger scores):
julia> report(mach).scores
5-element Vector{Float64}:
6.711689156091755e-7
-6.740101976655081e-7
-6.711632439648446e-7
-6.743015858874887e-7
-6.745393717880104e-7
## scores for new data:
Xnew = MLJ.table(rand(rng, 2, 3))
julia> transform(mach, rand(rng, 2, 3))
2-element Vector{Float64}:
-6.746293022511047e-7
-6.744289265348623e-7
Generating probabilistic predictions of outlierness
Continuing the previous example:
using OutlierDetection
pmodel = ProbabilisticDetector(model)
pmach = machine(pmodel, X) |> fit!
## probabilistic predictions on new data:
julia> y_prob = predict(pmach, Xnew)
2-element UnivariateFiniteVector{OrderedFactor{2}, String, UInt8, Float64}:
UnivariateFinite{OrderedFactor{2}}(normal=>1.0, outlier=>9.57e-5)
UnivariateFinite{OrderedFactor{2}}(normal=>1.0, outlier=>0.0)
## probabilities for outlierness:
julia> pdf.(y_prob, "outlier")
2-element Vector{Float64}:
9.572583265925801e-5
0.0
## raw scores are still available using `transform`:
julia> transform(pmach, Xnew)
2-element Vector{Float64}:
9.572583265925801e-5
0.0
Outlier classification using a probability threshold:
Continuing the previous example:
dmodel = BinaryThresholdPredictor(pmodel, threshold=0.9)
dmach = machine(dmodel, X) |> fit!
julia> yhat = predict(dmach, Xnew)
2-element CategoricalArrays.CategoricalArray{String,1,UInt8}:
"normal"
"normal"
User-defined kernels
Continuing the first example:
k(x1, x2) = x1'*x2 ## equivalent to `LIBSVM.Kernel.Linear`
model = OneClassSVM(kernel=k)
mach = machine(model, X) |> fit!
julia> yhat = transform(mach, Xnew)
2-element Vector{Float64}:
-0.4825363352732942
-0.4848772169720227
See also LIVSVM.jl and the original C implementation documentation. For an alternative source of outlier detection models with an MLJ interface, see OutlierDetection.jl.