[FEA]provide a way to parse and pack context_features and sequence_features

Open
#325 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
python, tensorflow
Domain
data-engineering, machine-learning

Research direction

Start by reviewing NVTabular's current Parquet support and TensorFlow's parse_sequence_example and parse_single_sequence_example references. Determine how nested SequenceExample-like data is represented, then evaluate the proposed parser callback or explicit context and sequence feature definitions. Done means the selected approach converts the nested data into tabular examples suitable for training.

Written by the indexing model from the issue text.

Description

enhancement

Is your feature request related to a problem? Please describe.
In many companies' recsys business, their raw recommender training data(no matter offline storage or streaming data) is usually organized in a SequenceExample way, which means in one example(stands for one user request context) it keeps all sequence feature sets(usually a list of items exposed to this user for this request, where click or not) and one single context feature set(usually the requesting user's feature and some context feature). They serialize, storage and transfer data in this way for better storage efficiency. And TensorFlow also offers methods like parse_sequence_example, parse_single_sequence_example to deserialize, make copies of context feature and pack them with every item of sequence feature into a single trainable example. Then every example is a complete one and can be shuffled with others. The total operation is like a flatMap.

Describe the solution you'd like
Current NVT supports Parquet, not TFRecord. I'm not sure how this SequenceExample like nested structure are being used in Parquet(how to serialize, deserialize?). But if Parquet supports this nested structure, maybe we can offer one of the below two ways to solve this problem.

  1. Let user define the parsing function and pass it into a flatMap function, which outputs the tabular like data.
  2. In a more strict way like tf.io.parse_sequence_example, let user define context feature name/type and sequence feature name/type, then let NVT parse the data into a tabular like data.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.

Dominant language
Python
Stars
1.2k
Forks
149
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA-Merlin/NVTabular

All issues in NVIDIA-Merlin/NVTabular

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.