agentscope-ai / agentscope-ai/Trinity-RFT

[FEATURE] Repetition filter operator for degenerate rollout responses

オープン
#620 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
701
フォーク
79
平均マージ
8時間 7分
マージ済み PR(30日)
1

説明

## Motivation

#470 invites new `ExperienceOperator` implementations, including "an operator to filter out
low-quality experiences based on predefined criteria". The existing filters cover reward thresholds
(`reward_filter`), zero-variance groups (`reward_std_filter`), DAPO dynamic sampling
(`dapo_dynamic_sampling`), truncated responses (`mask_response_truncated`) and invalid rewards
(`invalid_reward_filter`).

None of them looks at the *text* of the response. Repetition collapse is a common failure mode of
on-policy rollouts — especially for smaller models on long-horizon agent tasks: the policy keeps
emitting the same pattern instead of terminating. Those trajectories consume rollout budget and give
almost no learning signal, yet they currently enter the buffer like any other experience.

## Proposal

A `repetition_filter` operator that drops experiences whose response has a high duplicate n-gram ratio.

```yaml
explorer:
experience_pipeline:
operators:
- name: repetition_filter
args:
ngram_size: 4 # sliding window used to measure repetition
max_repeat_ratio: 0.5 # drop above this duplicate ratio
min_response_tokens: 32 # shorter responses are always kept (ratio too noisy)
```

- Only the model-generated part of the response is inspected: tokens before `prompt_length` are
ignored, and when `action_mask` is present (multi-turn experiences) observation tokens are ignored
as well.
- The response is dropped when `duplicate_ngrams / total_ngrams > max_repeat_ratio`; for example
`[7, 8, 9] * 20` scores `0.94` with `ngram_size=4`, while a diverse response scores `0.0`.
- Metrics: `filtered_count`, `skipped_short_count`, `skipped_empty_count`, `repeat_ratio/mean`,
`repeat_ratio/max`.
- Implemented on `ExperienceOperatorV1` (the async interface the operator guide asks new operators to
use), registered as `repetition_filter` in `EXPERIENCE_OPERATORS`, with unit tests covering the
ratio function, the filter, the `action_mask` path and registry creation.

## Question before I open the PR

1. Is a repetition/degeneracy filter in scope for the buffer, or would you rather see this handled
inside `Workflow`/`Algorithm`?
2. Dropping the experience is the simplest behaviour and matches the wording in #470. An alternative
is to keep the experience but zero its `action_mask` (like `mask_response_truncated` does), so the
rollout still counts for reward statistics. Which do you prefer?
3. Defaults: I picked `ngram_size=4`, `max_repeat_ratio=0.5`, `min_response_tokens=32`. Happy to
change them or drop the length guard.

I have the implementation and tests ready locally (CPU-only, `pytest tests/buffer/repetition_filter_test.py`
→ 11 passed) and will open the PR as soon as you confirm the direction.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。