[RMP] Merlin Models zoo benchmark

Problem:

It is not clear for customers which algorithm or model would provide the better accuracy.
There are many academic frameworks that provide some benchmarks for tasks like retrieval, CTR prediction, sequential and session-based recommendation. Such benchmarks are usually static and reported in papers. Those benchmarks in general are performed on small public datasets with very few features, not representative to the type and scale of real data hold by companies.

Goals:

• Becoming the de-facto public benchmark on RecSys models, while at the same time providing production-ready implementation of each of the benchmarked models
• Providing a way for customers to compare many different models using their own data

Requirements:

• Provide standard evaluation protocols for different RecSys tasks (e.g. Retrieval, CTR prediction, session-based recommendation).
• Provide standard metrics computation for different models across different frameworks
• Provide a flexible and standard way to compare the different models available
• Provide a number of datasets and preprocessing strategies that can be used for benchmarking
• Provide tooling for hyperparameter tuning optimization of the chosen models

Constraints:

• The benchmark requires large computational resources, as for each dataset and model a hyperparameter tuning session (e.g., 100 trials) will be needed for a fair comparison
• It could be initially an internal tool and open-sourced later

Starting Point:

• The BARS framework/benchmark ( https://openbenchmark.github.io/BARS/ ), which holds a public and live benchmark website. It can work as an inspiration for our benchmark