JuliaML / JuliaML/MLLabelUtils.jl

A Rare Label encoding, the combined all rare classes to one common label

Open
#5 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
31
Forks
13
PR merge metrics
No merged PRs in 30d

Description

In cases with highly unbalanced class distribution,
some classes occur in the training data so rarely, that it is better to ignore them.
In these cases, it would be useful to be able to set a threshold, and have all labels that occur less than this be mapped to a single label.

Here I will work with symbols for example clarity.

```julia
origlbls = [:A, :A, :A, :B, :B, :B, :C,:D, :A]

encoding, newlbls = relabel_rares(origlbls, threshold = 3, rarelabel=:X) #Signature for example only
@assert newlbls = [:A, :A, :A, :B, :B, :B, :X,:X, :A]
```

The data of which input labels are not rare, should be stored in the encoding, so it can be repeated.
Possibly it should also have a parameter to determine if new never before seen labels in the test data are an error, or just another rare label. (Possibly not, though).

This encoding would be chained with other encodings.

It would also go well with a method to filter in MLDataUtil,
for when it is permissible to exclude these rare labels from the training entirely.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.