multiple regressions in llama.cpp backend: `max_tokens` loops in streaming mode, first token duplicated
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 49.2k
- Forks
- 4.5k
- Avg merge
- 1d 3m
- Merged PRs (30d)
- 239
Description
I thought that surely by now #9298 would be fixed, so I foolishly decided to click 'reinstall' on my llama.cpp backend to get the latest and greatest. But:
-
A slightly different first-token-duplicate bug is either still there or newly introduced in the gRPC wrapper - only in streaming mode, and only for Qwen models AFAICT.
-
When
max_tokensis set and the token budget is exhausted during streaming, the endpoint resets and restarts generation rather than terminating withfinish_reason: "length". Each restart re-emits a usage chunk withcompletion_tokens==max_tokensandfinish_reason: null, then begins a fresh reasoning block from scratch. This repeats until the endpoint eventually emitsfinish_reason: "stop"— not a natural model stop from the backend, but apparently the wrapper giving up after several loops. Actual token consumption is therefore a multiple ofmax_tokens. Again, this is only in streaming mode.
Repro: POST /v1/chat/completions with "stream": true, "max_tokens": 10, (prompt doesn't matter, "count to 5" will do) and a thinking-capable model (e.g. Gemma4 MoE). For bonus points, a Qwen 3.5/3.6 model will also trigger the first bug.
I tried a cursory search in the issues (the state of which is what it is - not surprising given the target crowd) and apparently I'm the only one hitting this, which truly baffles me, especially since it must've been going on for weeks. Also @mudler: are there no basic regression tests for the backend wrappers?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the POST /v1/chat/completions request with stream=true and max_tokens=10 against a thinking-capable llama.cpp model, then trace the streaming path through the gRPC wrapper. Compare Qwen behavior for the duplicated first token and inspect what happens when the token budget is exhausted. Done means streaming emits the first token once and ends at the limit with finish_reason: "length" without restarting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, go
- Domain
- api, backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100