async grpo: trainer waits on an empty buffer forever after the trajectory collector dies
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
### Describe the bug
In async GRPO, when every `AsyncTrajectoryCollector` batch worker dies, the trainer never finds out. It polls the empty replay buffer forever:
```
Wait iteration 12996: buffer_size=0, step 0 ready=False
```
Observed on a 64-node run where all vLLM generation engines crashed on the first request after refit (separate root cause). The collector's batch workers raised after exhausting retries —
```
RuntimeError: Rollout batch failed to buffer prompt groups [0, 1, ..., 15]
❌ Error in NeMo-Gym batch worker (target_weight=1): ...
🧹 Released reservation for target weight 1
```
— but `grpo_train`'s wait loop has no failure channel from the collector, so the job idled for 3.7 hours (12,577 wait iterations, ~2,100 HTTP 500s from the dead engines) until the Slurm walltime killed it.
### Expected behavior
A fatal collector error (all batch workers dead, or N consecutive rollout batches failed with zero trajectories buffered) should propagate to the trainer and abort the run promptly. Optionally also a configurable no-progress watchdog (e.g. abort when the buffer stays empty for X minutes after run_start).
### Environment
64 nodes (16 train / 48 generation, non-colocated), vLLM 0.25.1 async engines, `async_grpo.enabled=true`.
Contributor guide
Assessment
This issue has not been assessed yet.