[FEA] Incremental Pre-processing
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- python
- Domain
- machine-learning
Research direction
No files or tests are mentioned. Start by tracing NVTabular's categorical preprocessing and mapping entry points, then review how existing encoded values are persisted and extended. Done means previous mappings remain stable and new categorical values receive contiguous ids suitable for appended embedding tables.
Written by the indexing model from the issue text.
Description
Is your feature request related to a problem? Please describe.
It is common in industry to train recommender systems models incrementally, i.e., taking a model trained with past data and fine-tuning with new data. In such cases, new values on existing categorical features need to be encoded as contiguous ids on top of the existing ones, whose embeddings will be appended to the pre-trained embedding tables.
Describe the solution you'd like
NVTabular should support incremental pre-processing, by keeping the previous mapping between raw values and encoded values for categorical features (so that they match the position of the pre-trained embeddings), and assigning new values as contiguous item ids.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 149
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from NVIDIA-Merlin/NVTabular
-
[DOC] Open
Difficulty 4/5 3-5 days Newbie friendliness 20/100
NVIDIA-Merlin/NVTabular#1898 · 1 comment ·
-
question
Difficulty 5/5 Over a week Newbie friendliness 25/100
NVIDIA-Merlin/NVTabular#1895 ·
-
question
Difficulty 4/5 3-5 days Newbie friendliness 25/100
NVIDIA-Merlin/NVTabular#1893 ·
-
question
Difficulty 5/5 Over a week Newbie friendliness 20/100
NVIDIA-Merlin/NVTabular#1892 · 1 reaction ·
-
question
Difficulty 4/5 3-5 days Newbie friendliness 35/100
NVIDIA-Merlin/NVTabular#1891 ·
All issues in NVIDIA-Merlin/NVTabular
Similar issues
-
area/auth bug comp/agent P3 platform/discord type/security
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NousResearch/hermes-agent#117848 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100