JuliaML / JuliaML/MLLabelUtils.jl

Suggestion: Restructure classify

Open
#30 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
31
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have been trying to extend the classify interface to binary NativeLabels. While figuring out the internals of classify, I have found some inconsistencies that make reasoning about the classify functionality difficult. I first summarize the inconsistencies, and then suggest a restructured Api to address them.

#### Inconsistencies:

* _`cutoff` field in LabelEnc Type:_ LabelEnc.ZeroOne currently is the only LabelEnc with a field `cutoff`. First, a cutoff generally is also be applicable to other LabelEncodings like MarginBased. However, I would argue that cutoff is not an inherent feature of the LabelEnc, and should therefore be external to it.
An example: computing an ROC curve for ZeroOne (or any other binary LabelEnc) requires to compute the false positive rate and true positive rate at all possible cutoffs. With the current implementation, this would require to modify or create a new LabelEnc, which I would argue does not make sense semantically.

This also leads to some special cases:

```julia
function classify(value::Number, lm::LabelEnc.ZeroOne{R}) where {R}
R(classify(value, lm.cutoff))
end
```

* _Default classify without encoding:_ In the current implementation `classify` without an encoding defaults to ZeroOne. To me this seems rather arbitrary, and I can't see the benefit of having this method overload. If there is some reason I do not see, maybe making things more explicit would help, i.e., replace

```julia
function classify(value::T, cutoff::Number) where {T<:Number}
value >= cutoff ? T(1) : T(0)
end
```

with

```julia
classify(value::Number, cutoff::Number) = classify(value, cutoff, LabelEnc.ZeroOne)
```

or to not provide this method at all.

* _Type dispatching:_ If I understand this correctly, dispatching on a Type argument, e.g., `classify(value::Number, ::Type{LabelEnc.ZeroOne})`, is like a default option for that LabelEnc. To me, these defaults may not always be intuitive, and for some encodings there cannot be a default, e.g., NativeLabels (what would be the poslabel in this case?). Also, the defaults may require more effort when reasoning about code.

* _Type inference on value argument:_ Currently, type inference if dispatching on the Type argument chooses the return type based on the type of the `value` argument. To me it is rather unintuitive why the type of `value`, which may be the result of some arbitrary scoring function, should determine the type of the classification result. Surprisingly, specifying the type for a LabelEnc does not work:

```julia
classify(0.6, LabelEnc.MarginBased{Float32}) # > MethodError
```

* _Function for broadcasting is semantically overloaded:_ There are two methods that take a vector as a first argument but they both mean different things. One time it is a broadcast helper, e.g., , the other time a single classify on a multivalued value vector. I currently don't see why one would need the broadcast helper function. Just provide the single-value classify, and leave the broadcasting to the caller.

* broadcast: `classify(values::AbstractVector{T}, cutoff::Number)`
* single classify multi-valued argument: `classify(values::AbstractVector, ::Type{<:LabelEnc.OneOfK})`

* _Minor naming convention_: Why is `lm` the naming convention for LabelEncodings variables? Would `lenc` be more suitable?

#### Suggested Api

Design principles for single value, binary classification:

* The label encoding is independent of classification, i.e., there are no special cases for a cutoff field
* In general, classify depends on an indicator function that decides whether a score value belongs to the positive class. It returns the poslabel/neglabel for the specified LabelEnc.
* The linear cutoff, i.e, a numeric value to distinguish between both classes, is a special case of this indicator function.
* Return type of classify must be explicit, i.e., no type inference based on `value` argument.
* Broadcasting is left to the caller

This would result in something like this:

```julia

function classify(value, cutoff::Function, lenc::BinaryLabelEncoding)
cutoff(value) ? poslabel(lenc) : neglabel(lenc)
end

function classify(value, cutoff::Number, lenc::BinaryLabelEncoding)
classify(value, x -> x > cutoff, lenc)
end

# some defaults
classify(value::Number, lenc::LabelEnc.ZeroOne) = classify(value, larger_than_cutoff(lenc.cutoff), lenc)
classify(value::Number, lenc::LabelEnc.MarginBased) = classify(value, signbit, lenc)
classify(value::Number, lenc::LabelEnc.NativeLabels{T,2,F}) where {T,F} = classify(value, 0.5, lenc)

# If there should still be "defaults" for non-initialized LabelEnc
classify(value::Number, lenc::Type{LabelEnc.ZeroOne{T}}) where T = classify(value, 0.5, lenc())
classify(value::Number, lenc::Type{LabelEnc.MarginBased{T}}) where T = classify(value, x -> !signbit(x), lenc())
```
With some modification this also works for vector-based classification, like OneOfK.

I would appreciate feedback on this first, before I put more implementation effort into this.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.