antirez / antirez/ds4

ds4-server: no reply-budget or forced think-close, so HTTP callers can hang where ds4-eval terminates

未关闭
#895 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
C
星标
22.3k
派生
2.1k
平均合并
1 天 3 小时
30 天内合并 PR
4

描述

## Summary

`ds4-eval` can force a `` close as the reply budget runs out. `ds4-server` cannot, and exposes no equivalent. The same model, on the same saturating question, terminates through `ds4-eval` and does not terminate over HTTP. The difference is the call path, not the model.

`ds4_eval.c:1216-1218`, with defaults at 1522-1524 and CLI flags at 1578-1580:

```c
int soft_limit_reply_budget; // 1024
int hard_limit_reply_budget; // 512
int soft_limit_think_close_rank; // 3
```

`grep -c "reply_budget\|think_close_rank" ds4_server.c` returns 0, and `./ds4-server --help` lists no equivalent.

## Measured cost

Overnight run, GLM-5.3-Flash Q2 on an M3 Max 128 GB, `--ctx 393216`, 14 hours, 415,425 tokens generated.

`ds4-eval` scored 71/92 with **21 failures that are all wrong answers and no non-answers** — the forced close means every question terminates. One question reached exactly 16,000 tokens, the cap, and still graded `PASSED`, because the harness made it conclude.

A separate 16-question suite driven over the HTTP API scored 14/16. The two failures are non-terminations: 60 minutes of silence each, no output. Across the night, **96,868 tokens — 23% of everything generated — produced zero usable characters**, all of it on the HTTP path.

Length is also a usable failure signal on this model. Separating passed from failed AIME items:

| | mean problem number | mean tokens generated |
|---|---|---|
| passed | 11.6 | 2,484 |
| failed | 14.5 | 8,477 |

The declared difficulty explains nothing — the problem numbers are nearly identical — while length separates the two populations by 3.4x, and half the AIME failures are exact 16,000-token saturations. Correlation rather than causation, since a hard question produces both length and error, but the counter is available in real time either way.

## Why it matters beyond a benchmark

Any agent talking to `ds4-server` over HTTP is exposed to non-termination, including `ds4-agent` itself when it goes through the API. On this model the budget is not spent on work, it is spent on persistence.

## Proposal

Port the three settings from `ds4_eval.c` to `ds4_server.c`, with the same defaults, as server flags and optionally as per-request fields. The logic already exists and is exercised; this is a plumbing change rather than a new mechanism.

Related: #879 asks for `--nothink` on `ds4-server`, which is the same shape of gap — the CLI and the eval harness expose generation controls that the server does not.

Happy to provide the traces, or to test a patch on the same 128 GB M3 Max setup.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。