NVIDIA-NeMo / NVIDIA-NeMo/Curator

Support batch sizes for shuffle stages/dedup stages.

Open
#1,332 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.8k
Forks
328
Avg merge
4d 5h
Merged PRs (30d)
30

Description

Is your feature request related to a problem? Please describe.
Currently duplicate identification (exact & fuzzy) prefers larger batches of data (1-2GB) to better saturate the GPU for both IO and downstream operations like shuffle etc. while removal might benefit from smaller blocks (256MiB-512MiB) to reduce memory ovehead on CPU cores.

In fuzzy dedup all stages after minhash use large blocksizes so minhash is the only stage where perf might suffer from smaller blocks.
In exact dedup the shuffle stage suffers a lot from smaller blocks.

Describe the solution you'd like
A hybrid approach where exact dedup supported larger batches (number of input tasks). It would still allow performing I/O on smaller batches at some cost of not saturating the GPU. But collecting and inserting multiple batches at once into the shuffler using the list of input tasks might help significantly with shuffle perf.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named. Start by locating the exact- and fuzzy-dedup shuffle stages and tracing how input tasks are grouped into batches; then identify the existing batch-size controls and related tests. Done means the relevant stages support the requested batch-size behavior without breaking deduplication or shuffle processing.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.