mudler / mudler/LocalAI

multiple regressions in llama.cpp backend: `max_tokens` loops in streaming mode, first token duplicated

Open
#9,716 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/llama.cpp bug unconfirmed waiting-from-reporter
Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

I thought that surely by now #9298 would be fixed, so I foolishly decided to click 'reinstall' on my llama.cpp backend to get the latest and greatest. But:

  1. A slightly different first-token-duplicate bug is either still there or newly introduced in the gRPC wrapper - only in streaming mode, and only for Qwen models AFAICT.

  2. When max_tokens is set and the token budget is exhausted during streaming, the endpoint resets and restarts generation rather than terminating with finish_reason: "length". Each restart re-emits a usage chunk with completion_tokens == max_tokens and finish_reason: null, then begins a fresh reasoning block from scratch. This repeats until the endpoint eventually emits finish_reason: "stop" — not a natural model stop from the backend, but apparently the wrapper giving up after several loops. Actual token consumption is therefore a multiple of max_tokens. Again, this is only in streaming mode.

Repro: POST /v1/chat/completions with "stream": true, "max_tokens": 10, (prompt doesn't matter, "count to 5" will do) and a thinking-capable model (e.g. Gemma4 MoE). For bonus points, a Qwen 3.5/3.6 model will also trigger the first bug.

I tried a cursory search in the issues (the state of which is what it is - not surprising given the target crowd) and apparently I'm the only one hitting this, which truly baffles me, especially since it must've been going on for weeks. Also @mudler: are there no basic regression tests for the backend wrappers?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the POST /v1/chat/completions request with stream=true and max_tokens=10 against a thinking-capable llama.cpp model, then trace the streaming path through the gRPC wrapper. Compare Qwen behavior for the duplicated first token and inspect what happens when the token budget is exhausted. Done means streaming emits the first token once and ends at the limit with finish_reason: "length" without restarting.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, go
Domain
api, backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.