NVIDIA-NeMo / NVIDIA-NeMo/Gym

Require manifests and version updates for new or changed workloads

Open
#2,971 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 21h
Merged PRs (30d)
318

Description

Problem

After the report-only period, new user-facing workloads must not add onboarding debt, and behavior changes must not be published under an unchanged workload version.

Depends on

  • #2001
  • #2969
  • #2970

Proposed change

Turn the changed-workload report into a blocking CI check while reusing the same impact-analysis and validation implementation.

For a new user-facing workload, require:

  • a valid manifest;
  • a valid verifier fixture;
  • successful validation and publication-readiness checks.

For an existing manifest-backed workload, compare the normalized resolved workload and component code identities with the base revision. If resolved behavior changes, require the manifest version to change. Failure output must name the workload and the changed direct or shared dependency.

Untouched legacy workloads remain allowed until their migration batch is selected.

Out of scope

  • Requiring immediate repository-wide migration.
  • Inferring whether a version change must be major, minor, or patch.
  • Running a full benchmark evaluation in the gate.

Acceptance criteria

  • A new user-facing workload without a valid manifest fails CI.
  • A new workload without a passing verifier fixture fails CI.
  • Directly changed manifest-backed workloads are validated.
  • Workloads affected by shared dependency changes are validated.
  • A resolved behavior change without a version update fails CI.
  • Failure output names the changed dependency.
  • Untouched legacy workloads remain allowed.
  • The blocking check uses the same result model as the report-only job.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing changed-workload report and trace its impact-analysis, validation implementation, and result model. Then review the dependent work in #2001, #2969, and #2970 before wiring the blocking CI check. Done means the listed acceptance cases pass, including new-workload validation, shared-dependency detection, version enforcement, named failures, and legacy-workload exceptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.