dotnet / dotnet/machinelearning-samples
Ranking example with TrainTest or CV splits
- Lingua principale
- PowerShell
- Stelle
- 4.7k
- Fork
- 2.7k
- Merge medio
- 2g 22h
- PR unite (30g)
- 1
Descrizione
We could use an example of how to split a ranking dataset into train/validate/test splits or CV/test. Currently, we have an example of how to use ranking using pre-defined train/validate/test dataset splits.
The current example doesn't demonstrate how to correctly split a ranking dataset without leaking.
Ranking needs to use the `samplingKeyColumnName` to split the ranking datasets. Without this, there is [row-wise leakage](https://en.wikipedia.org/wiki/Leakage_(machine_learning)#Training_example_leakage) as the set of candidate query results from a single query will be mixed in all of the training/validate/test sets. All of the results from single query must be within a single split/cv-fold.
The ranking `GroupId` (also commonly called `QueryId`/`QID`) column is often used as the `samplingKeyColumn`. Though other columns are often used to ensure the metrics are representative of how the model will perform when in production.
We may want to mention that the `samplingKeyColumnName` is best a string type so the value gets automatically hashed. In our splitter code, if the columns is an `Int32`/`Int64`, the value gets normalized to `0..1` and if it's a `Float` the value is directly used, then for both, the value directly mapped to the splits `trainset = 0..0.799` & `testSet = 0.80..1` ([code](https://github.com/dotnet/machinelearning/blob/f87a3bbd8adb12934dae0a0060813ce9b7500664/src/Microsoft.ML.Data/DataLoadSave/DataOperationsCatalog.cs#L559-L586)). This has the assumption that the input `Int`/`Float` are IID and uniformly distributed / randomly ordered; most QIDs that are INTs are not uniform and are order by time. The end result is the user tends to shoot themselves in the foot (which, I've done too many times for ranking) by making very unbalanced and/or biased splits / folds. More info: [[1](https://github.com/dotnet/machinelearning/issues/2487#issuecomment-462125732), [2](https://github.com/dotnet/machinelearning/issues/3711)]. Easier route is just to recommend `samplingKeyColumnName` be a string column (or modify our code to always hash).
We should be able to quickly make a ranking TrainTest or CV example using AutoML's CodeGen.
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia individuando l'esempio di ranking esistente e l'entry point di AutoML CodeGen menzionato nell'issue. Aggiungi un esempio TrainTest o CV che mantenga i risultati di ogni query in un singolo split o fold e documenta le indicazioni relative a samplingKeyColumnName, inclusa la raccomandazione di usare il tipo string. Il lavoro è completato quando l'esempio dimostra split di ranking sicuri dal leakage e le indicazioni sono chiare.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- csharp, machine-learning
- Ambito
- documentation, machine-learning
- Tipo di issue
- Documentazione
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 35/100