antirez / antirez/ds4

ds4-server reasoning_content deltas fragmented mid-token at arbitrary byte offsets (OpenAI-compatible streaming)

Offen
#685 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
22.3k
Forks
2.1k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
4

Beschreibung

## Summary
Streaming `/v1/chat/completions` emits `reasoning_content` deltas split at arbitrary byte offsets — mid-token, character-level — instead of whole tokens. Reassembled reasoning text is correct and the final `content` answer streams intact, so it's cosmetic for reassembly. But any client that consumes thinking deltas individually (reasoning-length estimation, per-token UIs, signature/checksum checks) gets garbage units, and every delta carries SSE+JSON overhead (wasted tokens).

## Environment
- ds4-server from antirez/ds4, DeepSeek V4 Flash GGUF (`ds4flash.gguf`)
- Launch: `ds4-server --ctx 262144 --port 8181 --model ./ds4flash.gguf`
- Client: live curl to `127.0.0.1:8181/v1/chat/completions`, `stream: true`

## Reproduction / observed
Live request to the local endpoint shows the reasoning stream fragmented at character level:
```
"1. T" → "he use" → "r a" → "sk" → "ed " → "to \"sa" → "y he" → ...
```
Every delta carries empty `"signature":""`. Reassembled text is correct ("1. The user asked to say hello..."); final answer `content` streams intact.

## Expected
Reasoning deltas aligned to token boundaries (one delta per decoded token), mirroring the `content` path.

## Root cause (in ds4-server, not Goose)
`openai_sse_stream_update()` in `ds4_server.c`, `OPENAI_STREAM_THINKING` branch (~line 6385):
```c
const size_t hold = strlen("") - 1;
limit = raw_len > hold ? raw_len - hold : st->emit_pos;
limit = utf8_stream_safe_len(raw, st->emit_pos, limit, false); // UTF-8 safe only
sse_chat_delta_n(fd, r, id, "reasoning_content", raw + st->emit_pos, limit - st->emit_pos);
```
Delta boundaries = accumulated text minus a 7-byte hold-back, clamped only to UTF-8 codepoint boundaries — never token-aligned. Identical shape in the Responses path (~7094) and Anthropic path (~7971). Regression test `test_openai_chat_stream_splits_reasoning_without_tools` (~13973) locks in current behavior.

## Impact
- Clients parsing thinking deltas individually (agent harnesses, reasoning-length counters, stream UIs) see mid-token garbage.
- Per-delta JSON overhead inflates output; "wastes tokens" vs whole-token deltas.
- Harmless for plain chat reassembly.

## Related issues (routing family — distinct from this granularity bug)
- #13, #509, #524, #665 (thinking→reasoning_content routing; #509 explicitly states streaming is "correct" re: labeling)
- #678 (close-tag handling in streaming TEXT mode)
- #681 (don't stop generation inside a thinking block)
- #167, #332, #612, #318 (unclosed/leaked thinking boundaries)

## Suggested fix
Accumulate reasoning per decode step and flush one delta per token (as `content` does), or hold back until the next token boundary; keep `utf8_stream_safe_len` as a floor, not the boundary rule.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.