NVIDIA-NeMo / NVIDIA-NeMo/Curator

[Docs] Align global pipeline execution concepts with current backends

Open
#2,161 0 comments 0 reactions 1 assignee View on GitHub

@lbliii is already working on this.

Since Jul 2, 2026.

documentation needs-follow-up r1.3.0
Dominant language
Python
Stars
1.8k
Forks
328
Avg merge
4d 5h
Merged PRs (30d)
30

Description

Parent tracker: NVIDIA-NeMo/Curator#2118

Goal

Align the global pipeline, resource, task, and migration concepts with the current Ray Data, Xenna, and Ray Actor Pool APIs documented for 26.07.

Problems found

  • about/concepts/resource-allocation.mdx imports removed modules: nemo_curator.stages.core and nemo_curator.stages.function_definitions.
  • about/concepts/auto-balancing.mdx presents dynamic worker balancing as universal and says manual sizing is unnecessary, although behavior is backend-specific.
  • about/concepts/throughput.mdx uses the nonexistent RaySlurmClient name instead of SlurmRayClient and repeats universal autoscaling claims.
  • The Dask-to-Ray migration FAQ says pipelines cannot fan out and stages cannot return multiple tasks, while ProcessingStage supports list outputs.
  • The experimental-executor page links to /reference/infrastructure/execution-backends instead of /reference/infra/execution-backends.
  • Framework-owned task IDs, deterministic lineage, output cardinality, NoneTask, and FailedTask are buried under resumability even though they apply to every custom stage.

Timing

Correctness fixes are release scope and should land before the 26.07 docs publication tag. A new task-lifecycle concept page may land in the same pass or be split into a clearly tracked follow-up if schedule requires.

Acceptance criteria

  • Replace removed imports with current ProcessingStage, Resources, and decorator paths.
  • Scope auto-balancing claims by executor and link to stage worker sizing.
  • Replace RaySlurmClient with SlurmRayClient and verify all throughput examples.
  • Correct the migration FAQ's fan-out, output-cardinality, worker-sizing, and recovery statements.
  • Fix the execution-backends route.
  • Add a backend decision table connecting resource allocation, worker sizing, runtime environments, and executor selection.
  • Add or plan a universal task-lifecycle/lineage concept and keep API/reference pages concise through cross-links.
  • Validate every changed Python import against the source tree and run Fern validation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.