[RMP] GTC Batch Email and Web Recommender

Problem:

We have an existing way to generate recommendations for GTC emails, but the process is fairly manual. Before the next GTC, we want to improve the structure of the recommender and automate the process to make this easier to use and maintain.

Goal:

• We want to use NVTabular as much as possible
• We want to frame the process of generating recs as Merlin Systems DAG
• We want to schedule this process with some kind of scheduling tool (e.g. Airflow)
• We want to create a template that customers can follow

New Functionality

• Dataloaders

  • (Nice to have) Make it possible to set the random seed

• NVTabular

  • Evaluate what feature processing operations required dropping down to pandas/cudf
  • Write new operators in NVT to cover those use cases (or implement as custom operators)
  • Take a look at NVT cross-joins to see if they can support what's needed here (instead of dropping down to cudf)
  • Investigate memory usage for processing text

• Systems

  • Categorical attribute-based filtering
  • Date/timestamp based filtering
  • Provide a way to pin certain items or categories of items into a particular list position
  • Provide a way to promote or boost items with an ordering operator
  • Review the existing pipeline to see what other new operators we might want to create

• Docs/Examples

  • Decide what format we'd like share the template in

Constraints:

• We'd like to use a released version of Merlin
• We don't have full access to the environment where the automated version of this recommender runs (have to align with another team)

Starting Point: