NVIDIA-Merlin / NVIDIA-Merlin/Merlin

[RMP] Quick-start for ranking models training pipeline

Open
#827 1 comment 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

roadmap
Dominant language
Python
Stars
907
Forks
129
No merged PRs in 30d

Description

Problem:

Merlin provides documentation and a number of example notebooks on how to use tools like NVTabular, Dataloader and Merlin Models. In order to build a pipeline for training and evaluation purposes, a Data Scientist needs to analyze that material, copy-and-paste code snippets demonstrating the API and glue that code together to implement scripts for experimentation and benchmarking.
It might also not be clear to the users the advanced API options featured by Merlin Models that can be mapped as a hyperparameter, and potentially improve models accuracy.

Goal:

This RMP provides a Quick-start for building ranking models training pipelines.
It addresses the ranking models part of this larger RMP NVIDIA-Merlin/models#732, in particular the steps 4-7 of the Data Scientist journey when experimenting with Merlin Models.

The Quick start for ranking is composed by:

Template scripts
  • Generic template script for preprocessing
  • Generic template script for training ranking models, exposing the main hyperparameters for ranking models .
    It includes support to ranking models like DLRM, DCN-v2, DeepFM, Wide&Deep and multi-task learning (MTL) like MMOE, CGC and PLE.
Documentation
  • Documentation of the scripts command line arguments
  • Documentation of best practices learned from our experimentation:
    • Hyperparameter tuning: search space, most important hyperparameters and best hparams for TenRec public dataset for STL and MTL models
    • Intuitions of API options (building blocks, arguments) that can improve models accuracy

Constraints:

  • Preprocessing - The pre-processing template notebook will perform some basic feature encoding for categorical (e.g. categorify) and continuous variables (e.g. standardization). The customer can expand the template with advanced preprocessing ops demonstrated in our examples.
  • Training - The training and evaluation script for Merlin Models should be totally configurable, taking as input the parquet files and schema, and a number of hyperparameters exposed via command line arguments. The output of this script should be the evaluation metrics, logged as a CSV file and also to Weights&Biases.

Starting Point:

The ranking training scripts we have developed for the MTL research project.

Tasks:

PR: https://github.com/NVIDIA-Merlin/models/pull/988

Tenrec dataset experiments
  • NVIDIA-Merlin/Merlin#680
  • NVIDIA-Merlin/Merlin#681
Documentation
  • NVIDIA-Merlin/Merlin#666
    • Create the main quick-start for ranking documention
    • Create a document explaining all arguments for preprocessing.py and ranking_train_eval.py
    • Create a document with the best practices for preprocessing
    • Create a document with the best practices for building and training ranking models
    • Create a document with the best practices for hypertuning and with the results of the TenRec experiments (in progress)
    • Create a tutorial on using Quick-start scripts with W&B Sweeps for HPO
Deployment and inference with Triton
Testing
  • NVIDIA-Merlin/Merlin#667
Blog post
  • #928

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.