agentscope-ai / agentscope-ai/Trinity-RFT
[FEATURE] Repetition filter operator for degenerate rollout responses
- Ngôn ngữ chính
- Python
- Star
- 701
- Fork
- 79
- Merge trung bình
- 8 giờ 7 phút
- Pull request đã merge (30 ngày)
- 1
Mô tả
## Motivation
#470 invites new `ExperienceOperator` implementations, including "an operator to filter out
low-quality experiences based on predefined criteria". The existing filters cover reward thresholds
(`reward_filter`), zero-variance groups (`reward_std_filter`), DAPO dynamic sampling
(`dapo_dynamic_sampling`), truncated responses (`mask_response_truncated`) and invalid rewards
(`invalid_reward_filter`).
None of them looks at the *text* of the response. Repetition collapse is a common failure mode of
on-policy rollouts — especially for smaller models on long-horizon agent tasks: the policy keeps
emitting the same pattern instead of terminating. Those trajectories consume rollout budget and give
almost no learning signal, yet they currently enter the buffer like any other experience.
## Proposal
A `repetition_filter` operator that drops experiences whose response has a high duplicate n-gram ratio.
```yaml
explorer:
experience_pipeline:
operators:
- name: repetition_filter
args:
ngram_size: 4 # sliding window used to measure repetition
max_repeat_ratio: 0.5 # drop above this duplicate ratio
min_response_tokens: 32 # shorter responses are always kept (ratio too noisy)
```
- Only the model-generated part of the response is inspected: tokens before `prompt_length` are
ignored, and when `action_mask` is present (multi-turn experiences) observation tokens are ignored
as well.
- The response is dropped when `duplicate_ngrams / total_ngrams > max_repeat_ratio`; for example
`[7, 8, 9] * 20` scores `0.94` with `ngram_size=4`, while a diverse response scores `0.0`.
- Metrics: `filtered_count`, `skipped_short_count`, `skipped_empty_count`, `repeat_ratio/mean`,
`repeat_ratio/max`.
- Implemented on `ExperienceOperatorV1` (the async interface the operator guide asks new operators to
use), registered as `repetition_filter` in `EXPERIENCE_OPERATORS`, with unit tests covering the
ratio function, the filter, the `action_mask` path and registry creation.
## Question before I open the PR
1. Is a repetition/degeneracy filter in scope for the buffer, or would you rather see this handled
inside `Workflow`/`Algorithm`?
2. Dropping the experience is the simplest behaviour and matches the wording in #470. An alternative
is to keep the experience but zero its `action_mask` (like `mask_response_truncated` does), so the
rollout still counts for reward statistics. Which do you prefer?
3. Defaults: I picked `ngram_size=4`, `max_repeat_ratio=0.5`, `min_response_tokens=32`. Happy to
change them or drop the length guard.
I have the implementation and tests ready locally (CPU-only, `pytest tests/buffer/repetition_filter_test.py`
→ 11 passed) and will open the PR as soon as you confirm the direction.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.