JuliaML / JuliaML/MLLabelUtils.jl
A Rare Label encoding, the combined all rare classes to one common label
- Dominant language
- Julia
- Stars
- 31
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
In cases with highly unbalanced class distribution,
some classes occur in the training data so rarely, that it is better to ignore them.
In these cases, it would be useful to be able to set a threshold, and have all labels that occur less than this be mapped to a single label.
Here I will work with symbols for example clarity.
```julia
origlbls = [:A, :A, :A, :B, :B, :B, :C,:D, :A]
encoding, newlbls = relabel_rares(origlbls, threshold = 3, rarelabel=:X) #Signature for example only
@assert newlbls = [:A, :A, :A, :B, :B, :B, :X,:X, :A]
```
The data of which input labels are not rare, should be stored in the encoding, so it can be repeated.
Possibly it should also have a parameter to determine if new never before seen labels in the test data are an error, or just another rare label. (Possibly not, though).
This encoding would be chained with other encodings.
It would also go well with a method to filter in MLDataUtil,
for when it is permissible to exclude these rare labels from the training entirely.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.