NVIDIA-NeMo / NVIDIA-NeMo/Curator
[Docs] Align global pipeline execution concepts with current backends
Open
@lbliii is already working on this.
Since Jul 2, 2026.
documentation
needs-follow-up
r1.3.0
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Parent tracker: NVIDIA-NeMo/Curator#2118
Goal
Align the global pipeline, resource, task, and migration concepts with the current Ray Data, Xenna, and Ray Actor Pool APIs documented for 26.07.
Problems found
about/concepts/resource-allocation.mdximports removed modules:nemo_curator.stages.coreandnemo_curator.stages.function_definitions.about/concepts/auto-balancing.mdxpresents dynamic worker balancing as universal and says manual sizing is unnecessary, although behavior is backend-specific.about/concepts/throughput.mdxuses the nonexistentRaySlurmClientname instead ofSlurmRayClientand repeats universal autoscaling claims.- The Dask-to-Ray migration FAQ says pipelines cannot fan out and stages cannot return multiple tasks, while
ProcessingStagesupports list outputs. - The experimental-executor page links to
/reference/infrastructure/execution-backendsinstead of/reference/infra/execution-backends. - Framework-owned task IDs, deterministic lineage, output cardinality,
NoneTask, andFailedTaskare buried under resumability even though they apply to every custom stage.
Timing
Correctness fixes are release scope and should land before the 26.07 docs publication tag. A new task-lifecycle concept page may land in the same pass or be split into a clearly tracked follow-up if schedule requires.
Acceptance criteria
- Replace removed imports with current
ProcessingStage,Resources, and decorator paths. - Scope auto-balancing claims by executor and link to stage worker sizing.
- Replace
RaySlurmClientwithSlurmRayClientand verify all throughput examples. - Correct the migration FAQ's fan-out, output-cardinality, worker-sizing, and recovery statements.
- Fix the execution-backends route.
- Add a backend decision table connecting resource allocation, worker sizing, runtime environments, and executor selection.
- Add or plan a universal task-lifecycle/lineage concept and keep API/reference pages concise through cross-links.
- Validate every changed Python import against the source tree and run Fern validation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.