dotnet / dotnet/machinelearning-samples

Ranking example with TrainTest or CV splits

Aperta
#832 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement New-Sample-App
Lingua principale
PowerShell
Stelle
4.7k
Fork
2.7k
Merge medio
2g 22h
PR unite (30g)
1

Descrizione

We could use an example of how to split a ranking dataset into train/validate/test splits or CV/test. Currently, we have an example of how to use ranking using pre-defined train/validate/test dataset splits.

The current example doesn't demonstrate how to correctly split a ranking dataset without leaking.

Ranking needs to use the `samplingKeyColumnName` to split the ranking datasets. Without this, there is [row-wise leakage](https://en.wikipedia.org/wiki/Leakage_(machine_learning)#Training_example_leakage) as the set of candidate query results from a single query will be mixed in all of the training/validate/test sets. All of the results from single query must be within a single split/cv-fold.

The ranking `GroupId` (also commonly called `QueryId`/`QID`) column is often used as the `samplingKeyColumn`. Though other columns are often used to ensure the metrics are representative of how the model will perform when in production.

We may want to mention that the `samplingKeyColumnName` is best a string type so the value gets automatically hashed. In our splitter code, if the columns is an `Int32`/`Int64`, the value gets normalized to `0..1` and if it's a `Float` the value is directly used, then for both, the value directly mapped to the splits `trainset = 0..0.799` & `testSet = 0.80..1` ([code](https://github.com/dotnet/machinelearning/blob/f87a3bbd8adb12934dae0a0060813ce9b7500664/src/Microsoft.ML.Data/DataLoadSave/DataOperationsCatalog.cs#L559-L586)). This has the assumption that the input `Int`/`Float` are IID and uniformly distributed / randomly ordered; most QIDs that are INTs are not uniform and are order by time. The end result is the user tends to shoot themselves in the foot (which, I've done too many times for ranking) by making very unbalanced and/or biased splits / folds. More info: [[1](https://github.com/dotnet/machinelearning/issues/2487#issuecomment-462125732), [2](https://github.com/dotnet/machinelearning/issues/3711)]. Easier route is just to recommend `samplingKeyColumnName` be a string column (or modify our code to always hash).

We should be able to quickly make a ranking TrainTest or CV example using AutoML's CodeGen.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia individuando l'esempio di ranking esistente e l'entry point di AutoML CodeGen menzionato nell'issue. Aggiungi un esempio TrainTest o CV che mantenga i risultati di ogni query in un singolo split o fold e documenta le indicazioni relative a samplingKeyColumnName, inclusa la raccomandazione di usare il tipo string. Il lavoro è completato quando l'esempio dimostra split di ranking sicuri dal leakage e le indicazioni sono chiare.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
csharp, machine-learning
Ambito
documentation, machine-learning
Tipo di issue
Documentazione
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.