NVIDIA-NeMo / NVIDIA-NeMo/RL

async grpo: trainer waits on an empty buffer forever after the trajectory collector dies

Open
#3,488 0 comments 0 reactions 1 assignee Claimed by @terrykong View on GitHub
bug Speed
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

### Describe the bug

In async GRPO, when every `AsyncTrajectoryCollector` batch worker dies, the trainer never finds out. It polls the empty replay buffer forever:

```
Wait iteration 12996: buffer_size=0, step 0 ready=False
```

Observed on a 64-node run where all vLLM generation engines crashed on the first request after refit (separate root cause). The collector's batch workers raised after exhausting retries —

```
RuntimeError: Rollout batch failed to buffer prompt groups [0, 1, ..., 15]
❌ Error in NeMo-Gym batch worker (target_weight=1): ...
🧹 Released reservation for target weight 1
```

— but `grpo_train`'s wait loop has no failure channel from the collector, so the job idled for 3.7 hours (12,577 wait iterations, ~2,100 HTTP 500s from the dead engines) until the Slurm walltime killed it.

### Expected behavior

A fatal collector error (all batch workers dead, or N consecutive rollout batches failed with zero trajectories buffered) should propagate to the trainer and abort the run promptly. Optionally also a configurable no-progress watchdog (e.g. abort when the buffer stays empty for X minutes after run_start).

### Environment

64 nodes (16 train / 48 generation, non-colocated), vLLM 0.25.1 async engines, `async_grpo.enabled=true`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.