ai-forever / ai-forever/deepagents-gigachat

LoopBreaker: restore nudge repeatability (regression shipped in 0.0.4)

Đang mở
#7 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
16
Fork
3
Merge trung bình
2 phút
Pull request đã merge (30 ngày)
1

Mô tả

## Problem

0.0.4 changed nudge de-duplication in `LoopBreakerMiddleware`:

- **0.0.3** — `_already_nudged()` walks the history backwards and **breaks at the first `AIMessage`**. The marker is only visible in the last block, so the budget nudge is **re-injected on every model turn** while the condition holds.
- **0.0.4** — `_nudge_count()` counts markers across the whole history, so each nudge fires **once per trajectory** (this is deliberate in #6 and marked by a comment).

The effect is not "the order arrives later", it is "the order stops arriving". A long trajectory that used to be pushed to finish by a repeating instruction now drifts until it hits the graph recursion limit.

## Evidence

Both measured on harness-bench-fast v0.16.0, 391 tasks, k=4 per arm, arms interleaved on one host and one stand.

**GigaChat-3.5-432B-A28B (IFT), native `run` runner** — the plus from the deterministic-output gate outweighs this, but the loss is visible: 0.0.4 loses 31 long multi-file tasks (renames, module splits, migrations), and step-limit hits go 2.2% -> 4.0%.

**GigaChat-3.1-10B reasoning (`opd` step 200), `run-openrouter`** — here nothing offsets it, because the three workspace-dependent middlewares are inert under that runner (it never calls `set_workspace_path`):

| arm | mean / 391 | recursion-limit hits |
|---|---:|---:|
| 0.0.3 | 292.8 (74.9%), sd 4.4 | 2.7% |
| PR #6 | 251.0 (64.2%), sd 8.3 | 23.7% |

Losses grow with task horizon (median steps of that task's successful attempts on 0.0.3): 0-8 steps −3.3 pp, 9-14 −10.2, 15-22 −22.4, 23+ −25.7.

**The threshold numbers are not the cause.** A fourth arm took PR #6 whole and reverted only the constants (24 -> 12 and 30 -> 16): 255.0/391 (65.2%), sd 5.8 — indistinguishable from PR #6 at 251.0. The tell is the distribution of tool rounds, not the mean:

| arm | 0-3 | 4-7 | 8-11 | 12-15 | 16+ | hit 80-step limit |
|---|---:|---:|---:|---:|---:|---:|
| 0.0.3 | 417 | 559 | 220 | **311** | 14 | 43 (2.7%) |
| PR #6 | 412 | 597 | 174 | 9 | 2 | 370 (23.7%) |
| thresholds reverted | 389 | 609 | 175 | **7** | 1 | 383 (24.5%) |

At the same threshold of 12 the pile past the threshold did not come back. In 0.0.3, 325 of 1521 recorded attempts (21.4%) reach 12 rounds and 145 of them succeed — 12.4% of all successes of that arm live in the band where the hard order has already fired. In PR #6 that band holds 11 attempts and 7 successes.

## Suggested fix

Keep `_nudge_count` for the grep/error-loop nudges (capping them at 2 is reasonable), but let the **budget** nudge re-inject while its condition holds — that is the safety valve. Whatever the shape, it needs its own measurement: the combination "PR #6 gate + repeatable nudge" has never been run.

## Workaround until then

Pin `deepagents-gigachat==0.0.3` when running the profile against a weaker model.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Start at LoopBreakerMiddleware and compare _already_nudged() with _nudge_count(), using the 0.0.3 and 0.0.4 behavior described in the issue. Keep grep/error-loop nudges capped while making the budget nudge repeatable when its condition holds, then measure the PR #6 gate combined with repeatable nudging on harness-bench-fast v0.16.0.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
backend
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Ít trao đổi
Độ rõ ràng
Khá rõ ràng
Mức phù hợp với người mới
62/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.