[GDPVal] Final stage emits certified metrics after exhausting retries, on any coverage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 349
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 324
Description
What happened
A GDPval baseline run completed with status: complete and emitted both
certified keys — comparison/stage_1/eval_elo 1585.06,
comparison/stage_1/normalized_elo 0.5425 — fitted on 86 of 220 tasks
(39%). Nothing in the run surfaced that.
Had it not been inspected by hand, that number would have been ingested as the
model's first-ever GDPval result. It is a model with no prior cell to
sanity-check against, and the fit also sits above the reference ladder's top
anchor, so there was no downstream signal that would have caught it either.
Mechanism
- 134 tasks each accumulated 3 failed attempts, exhausting
NEMO_GYM_MAX_ROLLOUT_ATTEMPTS=3. - 427 of those 428 failures were
timeout_exceeded— the 4h Slurm
partition reap, hit three times, on long trajectories that cannot finish
inside one leg. - With attempts exhausted the orchestrator stopped re-dispatching, declared the
stagecomplete, and fitted on whatever carried evidence: 86 rows. mean/expected_stage_row_countreads 220.0 against 86 actual. The
discrepancy is recorded and not acted on.
Two separable defects
(a) timeout_exceeded consumes rollout attempts. It reflects how much
walltime the leg had left, not anything about the task. Burning the 3-attempt
budget on reaps is what turned a slow run into a truncated one.
_is_terminal_failure() already treats the class as non-terminal; the attempt
cap should agree. A reaped rollout should be re-dispatched on the next leg
without charge.
(b) The final stage has no coverage floor. Non-final stages carry
partial_completion with min_success_fraction: 0.9, and that gate would have
rejected this outright. The final stage's plan is only
{num_tasks: 220, num_models: 4}, so a 39% sample and a 99% sample emit
identically-shaped, catalogable metrics. The stage-scoped keys are exactly the
ones leaves certify on, because the flat comparison/normalized_elo requires
220/220 — so the strict gate exists on the key nobody uses and no gate exists on
the key everybody uses.
Suggested direction
For (a): exclude timeout_exceeded from the attempt counter, or count attempts
only for classes _is_terminal_failure() considers terminal.
For (b): either allow partial_completion on the final stage, or refuse to emit
comparison/stage_N/* when the fitted row count falls materially short of
expected_stage_row_count — the values are already both present at emit time.
Failing loudly is clearly better than publishing a 39% fit that looks identical
to a complete one.
Evidence
HSG, 2026-09-12/13, invocation 9016a2e60ff492cb, four 4h legs. Attempt
histogram 1:142, 2:134. Failure classes 427 timeout_exceeded, 1 other.
A second run in the same sweep (c71abd45d04b4210) is on the same trajectory
with 134 timeout_exceeded and 90/220 cached, so this is reproducible rather
than a one-off.
Related: #3336 (a walltime reap landing in a non-waivable class by chance).
Both come from the same place — reap-shaped failures being treated as though
they said something about the rollout.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at _is_terminal_failure() and the rollout attempt-counting path to trace how timeout_exceeded failures consume the retry budget. Then inspect final-stage metric emission, including partial_completion and expected_stage_row_count. Done means reaped rollouts do not consume attempts and final-stage stage-scoped metrics are rejected when coverage is materially below the expected row count.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100