[Task] Set up integration tests for Merlin example notebooks

Problem:

• Currently, we have unit tests that cover partial dataset for the example notebooks. We cannot guarantee that example notebooks will be functional for the full dataset.

Goal:

• Integration tests that will confirm that example notebooks will be fully functional against the full data set.

Constraints:

• Previously, framework support for TF, PyT and HCTR was available only in NVTabular. With the recent architectural changes support must be made available in Merlin systems. Currently only TF support is available.

Starting Point:

• Create support for HugeCTR in Merlin Systems for inference
• Create support for Pytorch in Merlin Systems for inference
• Refactor integration tests in NVTabular, to remove all non feature processing.
• Create e2e example notebook for HugeCTR in merlin repo
• Create e2e example notebook for Pytorch in merlin repo
• Create CI for hugectr e2e notebook example in merlin repo
• Create CI for pytorch e2e notebook example in merlin repo
• Create jobs to run integration tests on different GPU architectures(?)
• Create jobs to run integration tests on different multi-GPU architectures(?)
• Create new system for reporting metrics to ASVDB
• Refactor integration tests to report captured metrics
• Setup system to display ASVDB metric (internally)
• Create DLs for ASVDB notifications - per repo (PIC, SIC, ??)
• Hook in appropriate DLs for Monitored metrics

All

• Create boilerplate for notebooks running integration tests using testbook
• https://github.com/NVIDIA-Merlin/Merlin/issues/285
• Add integration tests that use real data nightly

Merlin Models

• #212
• https://github.com/NVIDIA-Merlin/models/issues/482

Merlin Systems

• #213

Merlin

• #214