dotnet / dotnet/machinelearning

Improve SamplingKeyColumn documentation and usability

Open
#5,567 0 comments 1 reaction 0 assignees View on GitHub
area-Core documentation help wanted usability
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

The use `SamplingKeyColumn` is rather confusing. Perhaps we can improve it with documentation and runtime checks/warnings.

> @tasmektep: In your sample, you're using the [SamplingKeyColumn with your Label](https://github.com/tasmektep/DotNetMLSplit/blob/cf7ee532fd00221e5ad1fa420bd28678c4f52c85/DotNetMLSplit/Program.cs#L23 ) in it.
>
> Using your Label as your SamplingKeyColumn will cause all rows with the same Label value to be placed together in the same splits/folds (as you're seeing).
>
> Description from docs:
> > **SamplingKeyColumn**:
> > Name of a column to use for grouping rows. If two examples share the same value of the samplingKeyColumnName, they are guaranteed to appear in the same subset (train or test). This can be used to ensure no label leakage from the train to the test set. Note that when performing a Ranking Experiment, the samplingKeyColumnName must be the GroupId column. If null no row grouping will be performed.
> >
> > https://docs.microsoft.com/en-us/dotnet/api/microsoft.ml.dataoperationscatalog.traintestsplit?view=ml-dotnet
>
> You are likely thinking of the related, but inverse, concept of [Stratification](https://scikit-learn.org/stable/modules/cross_validation.html#stratified-k-fold) where the rows are evenly represented between the splits/folds. Stratification has some downsides causing it be less helpful.

_Originally posted by @justinormont in https://github.com/dotnet/machinelearning/issues/5563#issuecomment-749811593_

> @tasmektep: Keep posting issues that you run into. And thanks for posting your repro.
>
> Work on ML․NET side:
> * Warning -- Have the splitter warn when zero rows are present in a split, with a special warning if SamplingKeyColumn is used. In the same fix, we could warn of unbalanced splits/folds to help https://github.com/dotnet/machinelearning/issues/3711. Down side is the user would need to attach a logger to see the warning.
> * Documentation
> * Param hover -- Ensure the hover description for SamplingKeyColumn in Visual Studio is well worded to explain the concept, and perhaps mention what it does not do.
> * Samples/main docs -- Further explain the concept of SamplingKeyColumn, why its useful, and also what it does not do.
>

_Originally posted by @justinormont in https://github.com/dotnet/machinelearning/issues/5563#issuecomment-749822566_

In the param hover for `SamplingKeyColumn` in Visual Studio, mentioned above, we can also say to not put your `Label` column in `SamplingKeyColumn`.

It would be nice to automatically check this in `TrainTestSplit`, but the `SamplingKeyColumn` and `Label` aren't in the `TrainTestSplit` parameters together. The AutoML APIs can have this check as `SamplingKeyColumn` and `Label` are both in the parameters (and may want to throw an `ArgumentException` instead of warn).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.