[RMP] Quick-start RecSys pipeline and best practices guidance for training and evaluating retrieval and ranking models

This RMP is a re-write of NVIDIA-Merlin/Merlin#553

Problem:

Merlin platform provides libraries for building models training pipelines, including NVTabular, Dataloader and Merlin Models. When exploring Merlin, customers typically face different learning curves depending on their familiarity with RecSys. Here there is a typical funnel with steps that Data Scientists / ML Engineers get into to assess Merlin and its models with their data:

Customers new to RecSys

If new to RecSys or implementing their first RecSys pipeline, they need:

1. To perform some basic EDA to better understand the behavior of users in their dataset, in order to choose the most suitable features and models for their context.
2. To find some guideline on whether a single-stage (just retrieval) or two-stage (retrieval + ranking) would be more indicated for their dataset and which models would be suitable for those tasks and available data (e.g. MF x two-tower x session-based models for retrieval).

Customers new to Merlin

3. Check our example notebooks and documentation to better understand Merlin features and API
4. Create a preprocessing script/notebook for their dataset by copy-and-paste code snippets from examples and docs
5. Create the training script by checking our examples and docs to see how to glue the building blocks together
6. Train a few models and perform some early evaluation, to assess accuracy and performance (runtime / throughput)
7. Setup an offline evaluation with hyperparameter tuning to compare the different models available for the task (e.g. retrieval, ranking) with their data. This requires them to know enough from our API to be able to instantiate correctly the available models, to know how to set the advanced hyperparameters and to find a good search space for hyperparameters optimization. It might be hard for our customers to obtain a reasonable accuracy and performance without too much trial-and-error experimentation on their side.
8. Compare the best accuracy obtained with Merlin Models with their current RecSys model, implemented using other frameworks.

Merlin provides some examples and documentation for steps 3-5, but it is not straightforward to connect the different available pieces and to adapt them to customers data. And we provide very scarce code/documentation for steps 1-2 and 5-7.

Goal:

Provide a quick-start (template) pipeline and documented best practices to help customers to quickly build a pipeline that goes from basic EDA for RecSys datasets, to dataset preprocessing with NVTabular to models training and evaluation with Merlin Models.

Constraints:

Many of the above assets have been produced by our team for research, and can be used as a draft for our quick-start implementation for customers. Here are some constrains associated to steps listed above.

• Step 1 - EDA - The EDA (Exploratory Data Analysis) template notebook uses dask_cudf and is configurable, supporting basic generic statistics and plots like the frequency distributions of interactions by users, items, time, sessions. Specific analysis on features other than user_id, item_id, session_id and timestamp would have to be implemented by the customer with custom dataframe operations using dask_cudf.
  Starting point: RecSys Dataset Profiler
• Step 2 - Model decision flow - Creation a guideline document on how to choose the components of RecSys model pipeline based on the dataset
  Inspirations: Skit-learn and Tensorflow didactic flowcharts on ML models choosing
• Step 4 - Preprocessing - The pre-processing template notebook will perform some basic feature encoding for categorical (e.g. categorify) and continuous variables (e.g. standardization). The customer can expand the template with advanced preprocessing ops demonstrated in our examples.
• Steps 5 and 6 - Training - The training and evaluation script for Merlin Models should be totally configurable, taking as input the parquet files and schema, and a number of hyperparameters exposed via command line arguments. The output of this script should be the evaluation metrics, logged as a CSV file and also to Weights&Biases.
  Starting point: The configurable retrieval and ranking training scripts we have been in the research projects (latest versions here)
• Step 7 - Hyperparameter tuning - Here we would provide:
  • Documentation on how to setup a multi-node hyperparameter tuning using the training/evaluation script from Step 5 and Weights&Biases Sweeps
  • Analysis on the most important hyperparameters per model, with respect to the accuracy metrics, learned from our past hyperparameter tuning sessions
  • Search space for training retrieval and ranking models in YAML (W&B Sweeps format) with reasonable values (from our past hyperparameter tuning sessions)
  • Model Zoo - Comparison of training accuracy and performance (throughtput / runtime) for different retrieval and ranking models for handpicked public dataset. We would start with a single dataset (TenRec) for the first release and add more datasets over time.
• Step 8 - The cross-framework evaluation (#407) takes place here, allowing for computing metrics on persisted predictions from (black-box) retrieval and ranking models.

Tasks:

This RMP was divided in sub-RMPs

• Quick start for Ranking models (steps 4-7)
• Quick start for Retrieval models (steps 4-7)

NOTE: The following tasks are candidates to go to a separate RMP ticket targeted for RecSys beginners

• Automated dataset EDA (Step 1) - Package the RecSys Dataset Profiler notebook as part of the Quick-Start pipeline
• Model decision flow (Step 2) - Design a decision flow for choosing models for pipeline based on dataset characteristics and system requirements