NVIDIA-NeMo / NVIDIA-NeMo/DataDesigner
Implement the allocation-local Data Designer client worker
@andreatnvidia is already working on this.
Since Aug 24, 2026.
- Dominant language
- Python
- Stars
- 2.2k
- Forks
- 211
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 40
Description
Priority Level
High
Task Summary
Implement the allocation-local Data Designer client environment and worker that validates pinned inputs, invokes the public generation contract, and emits semantic attempt results for the Slurm runtime.
Technical Details & Implementation Plan
- Build the client environment from pinned, immutable package and artifact references.
- Reject editable installs, mutable source references, and distribution conflicts with the selected client image.
- Discover and validate required plugins and Data Designer configuration inside the allocation before generation starts.
- Invoke the public generation boundary from #851 without importing engine internals.
- Write bounded, structured progress and semantic client-result records for the runtime/state lane.
- Resolve managed assets through approved paths without serializing credentials.
Acceptance criteria
- The worker runs from built artifacts without a repository checkout.
- Dependency and plugin failures surface before generation and produce stable failure classifications.
- Generation uses only the public Data Designer contract and emits a validated semantic result.
- Client logs and persisted records contain no credentials or unredacted environment values.
- Tests cover dependency conflicts, missing plugins/assets, success, partial output, interruption, and normalized failures.
Out of scope
- Scheduler submission or allocation orchestration.
- Model-server startup and readiness.
- Winner publication, retry selection, collection, or benchmark analysis.
Investigation / Context
#851 defines the core public generation invocation boundary, #868 owns allocation sequencing, and #872 supplies deterministic test doubles. This task owns the production client environment and worker between those boundaries.
Agent Plan / Findings
Keep environment validation separate from generation invocation, and emit package-owned result records rather than leaking library or subprocess exceptions.
Dependencies
Depends on #851, the shared records in #873, and the fake infrastructure in #872. Integrates with #868 and #869.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.