dotnet / dotnet/machinelearning-samples

Ranking example with TrainTest or CV splits

Open
#832 0 comments 0 reactions 0 assignees View on GitHub
enhancement New-Sample-App
Dominant language
PowerShell
Stars
4.7k
Forks
2.7k
Avg merge
2d 22h
Merged PRs (30d)
1

Description

We could use an example of how to split a ranking dataset into train/validate/test splits or CV/test. Currently, we have an example of how to use ranking using pre-defined train/validate/test dataset splits.

The current example doesn't demonstrate how to correctly split a ranking dataset without leaking.

Ranking needs to use the `samplingKeyColumnName` to split the ranking datasets. Without this, there is [row-wise leakage](https://en.wikipedia.org/wiki/Leakage_(machine_learning)#Training_example_leakage) as the set of candidate query results from a single query will be mixed in all of the training/validate/test sets. All of the results from single query must be within a single split/cv-fold.

The ranking `GroupId` (also commonly called `QueryId`/`QID`) column is often used as the `samplingKeyColumn`. Though other columns are often used to ensure the metrics are representative of how the model will perform when in production.

We may want to mention that the `samplingKeyColumnName` is best a string type so the value gets automatically hashed. In our splitter code, if the columns is an `Int32`/`Int64`, the value gets normalized to `0..1` and if it's a `Float` the value is directly used, then for both, the value directly mapped to the splits `trainset = 0..0.799` & `testSet = 0.80..1` ([code](https://github.com/dotnet/machinelearning/blob/f87a3bbd8adb12934dae0a0060813ce9b7500664/src/Microsoft.ML.Data/DataLoadSave/DataOperationsCatalog.cs#L559-L586)). This has the assumption that the input `Int`/`Float` are IID and uniformly distributed / randomly ordered; most QIDs that are INTs are not uniform and are order by time. The end result is the user tends to shoot themselves in the foot (which, I've done too many times for ranking) by making very unbalanced and/or biased splits / folds. More info: [[1](https://github.com/dotnet/machinelearning/issues/2487#issuecomment-462125732), [2](https://github.com/dotnet/machinelearning/issues/3711)]. Easier route is just to recommend `samplingKeyColumnName` be a string column (or modify our code to always hash).

We should be able to quickly make a ranking TrainTest or CV example using AutoML's CodeGen.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.