NVIDIA-NeMo / NVIDIA-NeMo/Gym

[GDPVal] Walltime-teardown rollouts are classified 'legitimate', which is non-waivable

Open
#3,336 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.2k
Forks
349
Avg merge
1d 23h
Merged PRs (30d)
324

Description

What happened

During a full GDPval baseline sweep, one run reported legitimate: 152 on a
stage-2 batch. It was not 152 broken tasks. It was a single 4h Slurm walltime
reap whose in-flight rollouts Ray could not serialise; every one of them landed
in the legitimate bucket.

They recovered, but only incidentally: the rows were also non-terminal, so the
next auto-requeued leg re-dispatched them (the retry batch scoped correctly to
153 rows rather than redoing the stage).

Why it matters

legitimate is deliberately the one class the partial-completion policy will
never waive — waivable_failure_classes is [timeout_exceeded, transient], and
the comments in refs.yaml are explicit that "a real task failure must still
stop the stage". That is the right design.

The problem is that a purely infrastructural event can present as that class. If
a run had exited cleanly with those rows outstanding, the coverage gate would
have rejected a non-final stage over something that was never a task failure —
and per the orchestrator's own comment, a coverage rejection "requires an
explicit policy/data change", so nel resume would not have helped.

Note the sibling path is already handled correctly: the same reap, when it
surfaces as TaskPerAttemptTimeoutError, classifies to timeout_exceeded,
which is waivable.

Where

responses_api_agents/stirrup_agent/app.py, _classify_rollout_failure.

The function matches TaskPerAttemptTimeoutErrortimeout_exceeded,
TaskSampleSkipErrorskipped, the Ray kill family → kill_shaped, and
RayTaskErrorkill_shaped when there is no user-code frame. Everything else
falls through to return "legitimate" at the end.

A serialisation failure raised while the worker is being torn down at walltime
reaches that fallback. Semantically it belongs with kill_shaped or
timeout_exceeded: the rollout did not fail on its own merits, the allocation
ended underneath it.

Suggested direction

Make the fallback conservative for teardown-shaped exceptions rather than
defaulting them to the one non-waivable class. Either recognise the
serialisation error explicitly, or treat "exception raised after the walltime
deadline has passed" as timeout_exceeded regardless of type. The current
default optimises for not silently waiving real failures, which is correct, but
it means every unrecognised infrastructure fault inherits the strictest class.

Evidence

Observed 2026-09-12 on HSG, qwen-3.8-flash-next, invocation
c71abd45d04b4210, across legs 7100053 → 7104442 → 7104454. Found while running
a seven-model GDPval sweep for #3333.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in responses_api_agents/stirrup_agent/app.py at _classify_rollout_failure, and read the existing exception mappings and fallback behavior. Check the surrounding classifier coverage, then add regression coverage for the walltime-teardown serialization case while preserving the listed handling for timeout, skip, and Ray kill errors. Done means this infrastructure event no longer enters the non-waivable legitimate class.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
74/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.