THUDM / THUDM/slime

[Bug] Normalize reward advantages by explicit sample groups

Open
#2,230 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.5k
Forks
1.3k
Avg merge
5h 36m
Merged PRs (30d)
22

Description

Current limitation

RolloutManager._post_process_rewards can infer GRPO reward groups from the total sample count. When a custom rollout returns uneven numbers of samples per prompt, the fallback reshapes the rewards into one row and normalizes across the entire batch instead of within each prompt group.

For example, with group sizes 4, 3, and 4, a prompt whose three rewards are all 5.0 can receive advantages of -1.090909 instead of zero.

Expected behavior

Reward normalization should use each sample's explicit group_index and restore the normalized values in input order. If group identity is unavailable, uneven or mixed unidentified groups should not silently fall back to a global baseline. Singleton groups should produce zero when standard-deviation normalization is enabled.

Impact

Uneven custom rollout groups can receive incorrect advantages, changing the training signal.

Related pull request

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at RolloutManager._post_process_rewards and trace how reward groups are inferred and advantages are restored to input order. Reproduce the uneven group sizes 4, 3, and 4, then verify normalization uses explicit group_index, avoids a global baseline for unidentified mixed groups, and returns zero for singleton groups when standard-deviation normalization is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.