[RMP] Statistical Candidate Generation based on CoVisitation for two-stage recsys

Problem:

When the item catalog is very large (e.g. tens or hundreds of millions), companies typically implement a two-stage recsys pipeline, where (1) candidate generation models provide a number of candidate items for a given user or session and (2) a ranking model with more powerful features and architecture performs the final ranking.
The candidate generation models can be based on heuristics (e.g. most recent items), statistics (e.g. most popular, most co-visited items per session or per user) or machine learning (e.g. word2vec, MF, Two-Tower, YouTubeDNN), among others.
This RMP addresses the CoVisitation candidate generation algorithm, which has been very successful in the last competitions (Dressipi RecSys Challenge, Otto Kaggle competition). One example is the 3rd place solution from KGMON + Merlin team for Otto comp, where they used 20 different strategies to generate candidates from CoVisitation.

Goal:

This RMP proposes the implementation of CoVisitation candidate generation into Merlin.

New Functionality

NVTabular

1- CoVisitation Matrix Generation
Implement a CoVisitMatrix op in NVT backed by dataframe ops to compute a co-visit matrix (item_a, item_b, weight), allowing to configure Lambda ops for custom co-visit filter (i.e. what should be considered a co-visit, e.g. clicks, only purchases) and weighting strategy (i.e. how to weight multiple co-occurrence of items for the same user/session sequences and for multiple sequences).

Systems

2 - Candidate Generation from a CoVisitation Matrix

• Implement a CandidatesFromCoVisit op for NVTabular & Systems that takes one Co-visit matrix dataframe and a column with the sequence of session/user items, and optional Lambda op for custom weight aggregation. It outputs two list features (candidate item ids and candidate item scores)for each session/users based, sorted by the candidate scores

3 - Candidate Generation ensemble

• Create an op for Systems that allows combining candidate items from many candidate generation models (whether CoVisitation or models like MF,TwoTower), providing some option to choose how to score the candidates (e.g. frequency, sum of candidate scores, etc)

Starting Point:

• Description of the 3rd place solution from KGMON + Merlin team for Otto comp, by @cdeotte
  • Fast code to produce co-visit is with cuDF and GPU is here (anything named "GPU-XXX): link
  • Fast code to apply co-visit to user history is here (code named "model-709"): link
• Slack threads with more info about this proposal: here and here