NVIDIA-NeMo / NVIDIA-NeMo/Gym

[GDPVal] Final stage emits certified metrics after exhausting retries, on any coverage

Open
#3,341 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
324

Description

What happened

A GDPval baseline run completed with status: complete and emitted both
certified keys — comparison/stage_1/eval_elo 1585.06,
comparison/stage_1/normalized_elo 0.5425 — fitted on 86 of 220 tasks
(39%)
. Nothing in the run surfaced that.

Had it not been inspected by hand, that number would have been ingested as the
model's first-ever GDPval result. It is a model with no prior cell to
sanity-check against, and the fit also sits above the reference ladder's top
anchor, so there was no downstream signal that would have caught it either.

Mechanism

  1. 134 tasks each accumulated 3 failed attempts, exhausting
    NEMO_GYM_MAX_ROLLOUT_ATTEMPTS=3.
  2. 427 of those 428 failures were timeout_exceeded — the 4h Slurm
    partition reap, hit three times, on long trajectories that cannot finish
    inside one leg.
  3. With attempts exhausted the orchestrator stopped re-dispatching, declared the
    stage complete, and fitted on whatever carried evidence: 86 rows.
  4. mean/expected_stage_row_count reads 220.0 against 86 actual. The
    discrepancy is recorded and not acted on.

Two separable defects

(a) timeout_exceeded consumes rollout attempts. It reflects how much
walltime the leg had left, not anything about the task. Burning the 3-attempt
budget on reaps is what turned a slow run into a truncated one.
_is_terminal_failure() already treats the class as non-terminal; the attempt
cap should agree. A reaped rollout should be re-dispatched on the next leg
without charge.

(b) The final stage has no coverage floor. Non-final stages carry
partial_completion with min_success_fraction: 0.9, and that gate would have
rejected this outright. The final stage's plan is only
{num_tasks: 220, num_models: 4}, so a 39% sample and a 99% sample emit
identically-shaped, catalogable metrics. The stage-scoped keys are exactly the
ones leaves certify on, because the flat comparison/normalized_elo requires
220/220 — so the strict gate exists on the key nobody uses and no gate exists on
the key everybody uses.

Suggested direction

For (a): exclude timeout_exceeded from the attempt counter, or count attempts
only for classes _is_terminal_failure() considers terminal.

For (b): either allow partial_completion on the final stage, or refuse to emit
comparison/stage_N/* when the fitted row count falls materially short of
expected_stage_row_count — the values are already both present at emit time.
Failing loudly is clearly better than publishing a 39% fit that looks identical
to a complete one.

Evidence

HSG, 2026-09-12/13, invocation 9016a2e60ff492cb, four 4h legs. Attempt
histogram 1:142, 2:134. Failure classes 427 timeout_exceeded, 1 other.
A second run in the same sweep (c71abd45d04b4210) is on the same trajectory
with 134 timeout_exceeded and 90/220 cached, so this is reproducible rather
than a one-off.

Related: #3336 (a walltime reap landing in a non-waivable class by chance).
Both come from the same place — reap-shaped failures being treated as though
they said something about the rollout.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at _is_terminal_failure() and the rollout attempt-counting path to trace how timeout_exceeded failures consume the retry budget. Then inspect final-stage metric emission, including partial_completion and expected_stage_row_count. Done means reaped rollouts do not consume attempts and final-stage stage-scoped metrics are rejected when coverage is materially below the expected row count.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.