[Bug] Normalize reward advantages by explicit sample groups
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 1.3k
- Avg merge
- 5h 36m
- Merged PRs (30d)
- 22
Description
Current limitation
RolloutManager._post_process_rewards can infer GRPO reward groups from the total sample count. When a custom rollout returns uneven numbers of samples per prompt, the fallback reshapes the rewards into one row and normalizes across the entire batch instead of within each prompt group.
For example, with group sizes 4, 3, and 4, a prompt whose three rewards are all 5.0 can receive advantages of -1.090909 instead of zero.
Expected behavior
Reward normalization should use each sample's explicit group_index and restore the normalized values in input order. If group identity is unavailable, uneven or mixed unidentified groups should not silently fall back to a global baseline. Singleton groups should produce zero when standard-deviation normalization is enabled.
Impact
Uneven custom rollout groups can receive incorrect advantages, changing the training signal.
Related pull request
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at RolloutManager._post_process_rewards and trace how reward groups are inferred and advantages are restored to input order. Reproduce the uneven group sizes 4, 3, and 4, then verify normalization uses explicit group_index, avoids a global baseline for unidentified mixed groups, and returns zero for singleton groups when standard-deviation normalization is enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 66/100