THUDM / THUDM/slime

[Bug] shipped dynamic-sampling filters crash on fan-out groups (list[list[Sample]]) from multi-turn agent rollouts

Open Beginner friendly
#2,374 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.5k
Forks
1.3k
Avg merge
5h 36m
Merged PRs (30d)
22

Description

Summary

The shipped dynamic-sampling filters assume groups are list[Sample], but multi-turn agent rollouts fan out into list[list[Sample]] — a shape that generate_and_rm_group itself documents and that every other consumer of the group handles explicitly. Enabling --dynamic-sampling-filter-path with any custom-generate function that returns a list of samples crashes the rollout within the first training step:

AttributeError: 'list' object has no attribute 'get_reward_value'
Evidence
  1. The shipped filter calls sample.get_reward_value(args) for each element of samples, assuming samples: list[Sample]:
    • slime/rollout/filter_hub/dynamic_sampling_filters.py:9-11
  2. generate_and_rm_group documents both shapes (emphasis in the original docstring): "the group is list[Sample] for plain rollouts and list[list[Sample]] for the fan-out case":
    • slime/rollout/sglang_rollout.py:329-338
  3. The rollout loop itself defends against both shapes everywhere else — first-sample logging and final sorting both branch on isinstance(group[0], list):
    • slime/rollout/sglang_rollout.py:466-470, 494-497
  4. But call_dynamic_filter(dynamic_filter, args, group) at sglang_rollout.py:469 passes the raw group into the filter, which has no such branch.
Reproduction

Any rollout whose custom-generate function emits a list per trajectory (multi-turn agent fan-out, e.g. the pattern in examples/coding_agent_rl), with:

--dynamic-sampling-filter-path \
  slime.rollout.filter_hub.dynamic_sampling_filters.check_reward_nonzero_std_with_fallback

Minimal Python repro (no GPU):

import types
from slime.rollout.filter_hub.dynamic_sampling_filters import check_reward_nonzero_std
from slime.utils.types import Sample

args = types.SimpleNamespace(reward_key=None)
group = [[Sample(prompt="p", label="l", reward=0.2)],   # fan-out: list[list[Sample]]
         [Sample(prompt="p", label="l", reward=1.0)]]
check_reward_nonzero_std(args, group)
# AttributeError: 'list' object has no attribute 'get_reward_value'

Observed on slime main @ a3f50097, single-GPU H100, agentic GRPO run — reproduced at both N_SAMPLES=4 and N_SAMPLES=8.

Suggested fix

Handle the fan-out shape in the shipped filters, mirroring the pattern already used by the rollout loop:

def _entry_reward(args, entry):
    # fan-out fragments of one trajectory share the trajectory reward
    return entry[0].get_reward_value(args) if isinstance(entry, list) else entry.get_reward_value(args)

def check_reward_nonzero_std(args, samples: list, **kwargs):
    rewards = [_entry_reward(args, s) for s in samples]
    ...

We shipped a fan-out-safe variant locally as a workaround and are happy to send a PR if the maintainers agree with the approach (fragments of one trajectory sharing entry[0]'s reward).

Related
  • #2370 (same "agentic/multi-turn path is under-tested" family: final-step unconditional save)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with slime/rollout/filter_hub/dynamic_sampling_filters.py and compare its shipped filters with the fan-out shape documented in slime/rollout/sglang_rollout.py:329-338 and handled near lines 466-470 and 494-497. Run the minimal no-GPU reproduction from the issue, then exercise the filter through the dynamic-sampling path with both list[Sample] and list[list[Sample]] groups. Done means fan-out rollouts no longer raise AttributeError and plain rollouts continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.