NVIDIA-NeMo / NVIDIA-NeMo/Curator
Investigate `preview` vs `create` for `DataDesignerStage`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 327
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
For https://github.com/NVIDIA-NeMo/Curator/pull/1449, preview is being used to work with DataFrames pre-loaded by a Curator stage. We can look into switching over to create that generates durable artifacts on disk.
Context: preview is not meant for any meaningful larger generation. While it will let you specify a large number of rows to generate, it doesn't have the durability mechanisms built into create that lets you resume from checkpoints and scale across concurrent row groups. preview also exposes your workers vulnerable to OOMs, at the mercy of the size of each partition a Curator stage loads and the underlying Data Designer pipeline.
Thread: https://github.com/NVIDIA-NeMo/Curator/pull/1449#discussion_r2777992738
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with PR #1449 and its linked discussion to trace how DataDesignerStage currently uses preview and where create could replace it. Compare the durability, checkpointing, concurrency, and memory implications described in the issue. Done means reaching and documenting a clear decision about the switch, with the relevant implementation or follow-up identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100