antirez / antirez/ds4

ds4-server: no reply-budget or forced think-close, so HTTP callers can hang where ds4-eval terminates

Abierto
#895 1 comentario 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
22.3k
Forks
2.1k
Merge medio
1 d 3 h
PR fusionados (30 d)
4

Descripción

## Summary

`ds4-eval` can force a `` close as the reply budget runs out. `ds4-server` cannot, and exposes no equivalent. The same model, on the same saturating question, terminates through `ds4-eval` and does not terminate over HTTP. The difference is the call path, not the model.

`ds4_eval.c:1216-1218`, with defaults at 1522-1524 and CLI flags at 1578-1580:

```c
int soft_limit_reply_budget; // 1024
int hard_limit_reply_budget; // 512
int soft_limit_think_close_rank; // 3
```

`grep -c "reply_budget\|think_close_rank" ds4_server.c` returns 0, and `./ds4-server --help` lists no equivalent.

## Measured cost

Overnight run, GLM-5.3-Flash Q2 on an M3 Max 128 GB, `--ctx 393216`, 14 hours, 415,425 tokens generated.

`ds4-eval` scored 71/92 with **21 failures that are all wrong answers and no non-answers** — the forced close means every question terminates. One question reached exactly 16,000 tokens, the cap, and still graded `PASSED`, because the harness made it conclude.

A separate 16-question suite driven over the HTTP API scored 14/16. The two failures are non-terminations: 60 minutes of silence each, no output. Across the night, **96,868 tokens — 23% of everything generated — produced zero usable characters**, all of it on the HTTP path.

Length is also a usable failure signal on this model. Separating passed from failed AIME items:

| | mean problem number | mean tokens generated |
|---|---|---|
| passed | 11.6 | 2,484 |
| failed | 14.5 | 8,477 |

The declared difficulty explains nothing — the problem numbers are nearly identical — while length separates the two populations by 3.4x, and half the AIME failures are exact 16,000-token saturations. Correlation rather than causation, since a hard question produces both length and error, but the counter is available in real time either way.

## Why it matters beyond a benchmark

Any agent talking to `ds4-server` over HTTP is exposed to non-termination, including `ds4-agent` itself when it goes through the API. On this model the budget is not spent on work, it is spent on persistence.

## Proposal

Port the three settings from `ds4_eval.c` to `ds4_server.c`, with the same defaults, as server flags and optionally as per-request fields. The logic already exists and is exercised; this is a plumbing change rather than a new mechanism.

Related: #879 asks for `--nothink` on `ds4-server`, which is the same shape of gap — the CLI and the eval harness expose generation controls that the server does not.

Happy to provide the traces, or to test a patch on the same 128 GB M3 Max setup.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.