dotnet / dotnet/machinelearning
Getting unablanced (or empty) folds when running CV with a SamplingKeyColumn (such as when running CV with Ranking)
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
The change in KeyDataViewType to disallow unknown cardinality keys, fixed a silent bug in CV that caused the group column to not be used for stratification (causing label leakage between the training and test sets), but now that it is fixed there is a new bug.
CV now uses the GroupId column for stratification, but the values in the GroupId column may not be distributed evenly, causing all the examples to be in one fold, and the others would be empty.
For example: Assume that the values in the GroupId column are `10,11,...,19`, and that they are loaded with `TextLoader` as a key type column. The key type would be of cardinality 20, and if there are 2 folds, then CV will try to create one fold with all the examples with values `0,...,9` and the other fold with all the examples with values `10,...,19`.
Contributor guide
Assessment
This issue has not been assessed yet.