dotnet / dotnet/machinelearning

Getting unablanced (or empty) folds when running CV with a SamplingKeyColumn (such as when running CV with Ranking)

Open
#3,711 11 comments 0 reactions 0 assignees View on GitHub
area-Core bug Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

The change in KeyDataViewType to disallow unknown cardinality keys, fixed a silent bug in CV that caused the group column to not be used for stratification (causing label leakage between the training and test sets), but now that it is fixed there is a new bug.
CV now uses the GroupId column for stratification, but the values in the GroupId column may not be distributed evenly, causing all the examples to be in one fold, and the others would be empty.
For example: Assume that the values in the GroupId column are `10,11,...,19`, and that they are loaded with `TextLoader` as a key type column. The key type would be of cardinality 20, and if there are 2 folds, then CV will try to create one fold with all the examples with values `0,...,9` and the other fold with all the examples with values `10,...,19`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.