Batched server prefills can starve active decoders between quanta
- Lenguaje dominante
- C
- Estrellas
- 22.4k
- Forks
- 2.1k
- Merge medio
- 2 d 13 h
- PR fusionados (30 d)
- 5
Descripción
## Summary
With `--batched-session 3`, a long prefill can repeatedly reacquire the model between its nominal 128-token mixed-prefill quanta. Active decoders may consequently make no progress for the full duration of the prefill (60–70 seconds in the traces below).
This appears to be a scheduler fairness race rather than a Metal throughput or correctness problem: once all three sessions are decoding, aggregate throughput remains around 24–25 tok/s.
## Environment
- ds4 main at `0a7ad77` (`Fix batched server session recovery race`)
- Apple M5 Max, 128 GiB RAM
- Metal backend with Metal 4 tensor API enabled
- DeepSeek V4 Flash, quant 2
- OpenCode using the OpenAI-compatible chat/tool API
Launch command:
```sh
./ds4-server \
--ctx 200000 \
--prefill-chunk 2048 \
--batched-session 3 \
--kv-disk-dir /tmp/ds4-kv \
--kv-disk-space-mb 24576
```
Startup confirms the intended scheduler configuration:
```text
ds4-server: batched mode enabled resident_sessions=3 prefill_quantum=2048 mixed_prefill_quantum=128 decode_coalesce_us=2000
```
## Reproduction / evidence
One resident session was actively decoding while another began a resumed 5,361-token prefill. The decoder produced token 600 at `20:37:08`, then made no progress until the entire prefill finished at `20:38:19`:
```text
20:37:08 chat ctx=7469..7519:50 gen=600 THINKING decoding ...
20:37:08 chat ctx=164081..169442:5361 TOOLS prompt start
20:37:10 chat ... prefill chunk 128/5361
...
20:38:17 chat ... prefill chunk 5248/5361
20:38:19 chat ... prefill chunk 5361/5361
20:38:19 chat ... prompt done 70.785s
20:38:19 chat ctx=7519..7569:50 gen=650 THINKING decoding ...
```
The same pattern repeated immediately afterward with a 4,664-token prefill, causing another roughly 60-second decode pause.
## Expected behavior
When any generation is active, completing a bounded prefill quantum should hand the next model turn to a pending/active decoder before another prefill quantum can start. The bounded quantum should place a finite upper bound on decode latency.
## Root cause
`server_prefill_leave()` clears `model_busy` and broadcasts `model_cv`, but there is no explicit handoff to an active generator. The prefill worker immediately loops back into `server_prefill_enter()` and can reacquire `model_mu` before a generation worker reaches `server_eval_token()` and increments `decode_pending`.
The admission condition prioritizes an already-pending decode, but it does not reserve the next turn for an active generator that has not enqueued its next token yet. Repeated wins by the prefill thread turn the 128-token quantum into an effectively unbounded decode stall.
## Proposed fix
Use an explicit prefill-to-decode handoff:
1. On leaving a prefill quantum while `active_generations > 0`, set a scheduler handoff flag.
2. Block further prefill admission while that flag is set.
3. Clear it after the next decode batch completes, or when the active generation count reaches zero/shutdown begins.
I have a small local patch implementing this and a threaded regression test that verifies the prefill worker remains blocked until a decode completion releases the handoff.
## Validation
After applying the handoff locally:
- Server unit tests pass.
- The Metal server builds without warnings.
- A real OpenCode request plus two identical concurrent streamed requests occupied all three resident slots.
- The duplicate requests returned identical content, reasoning, finish reason, and token counts.
- During three-way decode, each stream progressed at approximately 7.5–9 tok/s, consistent with the expected 24–25 tok/s aggregate.
- No long prefill-induced decoder pause was observed.
I can submit the patch as a PR if this scheduling approach looks appropriate.
Guía de contribución
Línea de trabajo
Start by tracing server_prefill_enter() and server_prefill_leave() alongside server_eval_token(), then run the existing server unit tests and inspect the mentioned threaded regression test. Done means the next decode turn is reserved after a prefill quantum while active generations remain, the regression test proves prefill admission stays blocked until decode completion, and the server tests and build pass.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- c
- Área
- backend, performance
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Tranquilo
- Claridad
- Bien especificado
- Aptitud para principiantes
- 48/100