dotnet / dotnet/machinelearning-samples
Ranking example with TrainTest or CV splits
- Dominant language
- PowerShell
- Stars
- 4.7k
- Forks
- 2.7k
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 1
Description
We could use an example of how to split a ranking dataset into train/validate/test splits or CV/test. Currently, we have an example of how to use ranking using pre-defined train/validate/test dataset splits.
The current example doesn't demonstrate how to correctly split a ranking dataset without leaking.
Ranking needs to use the `samplingKeyColumnName` to split the ranking datasets. Without this, there is [row-wise leakage](https://en.wikipedia.org/wiki/Leakage_(machine_learning)#Training_example_leakage) as the set of candidate query results from a single query will be mixed in all of the training/validate/test sets. All of the results from single query must be within a single split/cv-fold.
The ranking `GroupId` (also commonly called `QueryId`/`QID`) column is often used as the `samplingKeyColumn`. Though other columns are often used to ensure the metrics are representative of how the model will perform when in production.
We may want to mention that the `samplingKeyColumnName` is best a string type so the value gets automatically hashed. In our splitter code, if the columns is an `Int32`/`Int64`, the value gets normalized to `0..1` and if it's a `Float` the value is directly used, then for both, the value directly mapped to the splits `trainset = 0..0.799` & `testSet = 0.80..1` ([code](https://github.com/dotnet/machinelearning/blob/f87a3bbd8adb12934dae0a0060813ce9b7500664/src/Microsoft.ML.Data/DataLoadSave/DataOperationsCatalog.cs#L559-L586)). This has the assumption that the input `Int`/`Float` are IID and uniformly distributed / randomly ordered; most QIDs that are INTs are not uniform and are order by time. The end result is the user tends to shoot themselves in the foot (which, I've done too many times for ranking) by making very unbalanced and/or biased splits / folds. More info: [[1](https://github.com/dotnet/machinelearning/issues/2487#issuecomment-462125732), [2](https://github.com/dotnet/machinelearning/issues/3711)]. Easier route is just to recommend `samplingKeyColumnName` be a string column (or modify our code to always hash).
We should be able to quickly make a ranking TrainTest or CV example using AutoML's CodeGen.
Contributor guide
Assessment
This issue has not been assessed yet.