[RMP] Performance improvements for Merlin Models

Problem:

Merlin Models TF API provides high- and low-level building blocks that makes it easy to build different DL architectures for retrieval and ranking models. Therefore, there has not been devoted yet much effort in optimizing its training performance.

Context

@vysarge has done a number of benchmark and profiling experiments using using synthetic data comparing Merlin Models DLRM with the JoC DLRM TF implementation. This RMP ticket aggregates tasks related to her findings and suggestions. The summary of her experiments results can be found in this spreadsheet (Nvidia internal only)

Goal:

Implement fixes and improvements that are able to improve Merlin Models performance and document the best practices for our users

Tasks in Merlin Models

• NVIDIA-Merlin/models#1033
• NVIDIA-Merlin/models#1034
• NVIDIA-Merlin/models#1035
• NVIDIA-Merlin/models#1036
• NVIDIA-Merlin/models#1037
• NVIDIA-Merlin/models#1038