[RMP] Ability to save a model for inference where the embeddings are saved separately as an npz file and the model is saved such that embeddings are an input.

Goal:
Customers want to be able to treat embeddings created through Merlin as pretrained embeddings at inference time.

Flesh out how customers get from our saving of embeddings as a merlin dataset to getting data into a data storage

Motivation:

• Enables serving your on a smaller GPU than used for training. Where the model is larger than the smaller GPU size.

Models:

• Export embeddings trained with Merlin Models as a Merlin dataset
• Notes: Currently possible in EmbeddingFeatures.embedding_table_dataset(), needs to be ported to the newer EmbeddingTable API). The Merlin dataset has already functionality to export a dataset to npz
• Change the input block of the model for serving to use pretrained embeddings instead of categorical features as inputs. Save the model in that format for serving through systems.
• Being able to remove the embedding table from the model when saving for inference, as the pre-trained embeddings will be fed by the dataloader
  Notes:
• New PyTorch API will allow extracting blocks from the model for serving. Tensorflow functionality to do this will need to be added.
• Question: Should that functionality in TF API be generic (extraction of any block from the model) in TF API too (probably a large refactory) or should we implement it initially just for this use case? @marcromeyn can help to answer this question, as he knows the effort to implement the block extraction in PyT API

Systems:

• Pretrained embedding operator using numpy (memmaps into memory and onto disk)
• Feast operator
• Evaluate the generalization and validate that we can pull features from Feast at inference time
• Evaluate performance of the different data access operators