NVIDIA-NeMo / NVIDIA-NeMo/RL

Rollout observability: GRPO uploads 1-2 rollouts per step to W&B

Open
#938 0 comments 0 reactions 0 assignees View on GitHub
research
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

Currently NeMo RL will [dump all the GRPO train rollouts to disk](https://github.com/NVIDIA-NeMo/RL/blob/main/nemo_rl/algorithms/grpo.py#L781) (thank you @terrykong for the pointer).

However, it would be great to have a more accessible way of getting rollout observability, namely in W&B. We should upload a small sample of rollouts to W&B (like the actual trajectories not just statistics). It can help catch sneaky and scary vllm issues like this one (found and shared by @ertkonuk) https://github.com/vllm-project/vllm/issues/17652

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.