NVIDIA-NeMo / NVIDIA-NeMo/DataDesigner
Compile Slurm configuration into deterministic execution plans
@andreatnvidia is already working on this.
Since Aug 24, 2026.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 211
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 40
Description
Priority Level
High
Task Summary
Resolve strict authored Slurm input and cluster profiles into an immutable, deterministic execution plan before any scheduler submission occurs.
Technical Details & Implementation Plan
- Load and validate authored run configuration and the selected cluster profile.
- Resolve Data Designer invocation inputs, deployments, images, resources, mounts, shard ranges, client placement, and output intent into package-owned plan records.
- Keep authored semantics separate from resolved scheduler/runtime facts.
- Produce stable plan serialization and digests suitable for pinning in run, shard, and attempt records.
- Validate cross-field resource, placement, path, and deployment constraints before submission.
- Provide pure planning entry points that require no Slurm commands or runtime services.
Acceptance criteria
- The same authored input, profile, and referenced artifacts produce identical plan bytes and digests.
- Invalid deployment, placement, resource, mount, shard, and destination combinations fail before submission.
- Plans contain all inputs required by serving, image, launcher/runtime, client-worker, state, and benchmark lanes.
- Plans contain no credentials, mutable source references, or unresolved environment-dependent defaults.
- Unit and golden tests cover single-node, multi-node, multi-deployment, sharded, and invalid configurations.
Out of scope
- Slurm submission or command parsing.
- Starting model services or the Data Designer client.
- State persistence, retry, collection, or benchmark execution.
Investigation / Context
This is the configuration-resolution and deterministic-plan lane in #850. #866 and #868 consume planner-supplied placement and resources but do not own authored configuration or plan compilation.
Agent Plan / Findings
Keep planning pure and deterministic so every side effecting lane consumes a reviewed immutable plan rather than re-resolving authored input.
Dependencies
Depends on the public contract in #851, the optional package foundations in #852 and #853, and the shared records in #873.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.