lmstudio-ai / lmstudio-ai/mlx-engine
Gemma 4 (26B-A4B) reasoning never terminates on MLX — fills max_tokens with reasoning_content, empty content (GGUF terminates fine)
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 133
- Avg merge
- 21h 6m
- Merged PRs (30d)
- 1
Description
**Summary**
On the MLX runtime, Gemma 4 26B-A4B reasoning **never terminates**: the model emits `reasoning_content` until it hits `max_tokens` and returns empty `content` (`finish_reason: "length"`). The *same* model on LM Studio's llama.cpp/GGUF backend reasons but **terminates normally** and returns content — so this non-termination appears MLX-engine-specific.
**Environment**
- LM Studio MLX engine `0.4.16+2`
- Model: `google/gemma-4-26b-a4b-qat` (MLX, Q4 QAT)
- macOS, Apple Silicon (M4 Pro)
**Observed (MLX)**
```json
"choices": [{ "message": { "content": "", "reasoning_content": "...~16k tokens..." }, "finish_reason": "length" }],
"usage": { "completion_tokens": 15999, "completion_tokens_details": { "reasoning_tokens": 15997 } }
```
`content` is empty; the entire budget is consumed by reasoning that never closes.
**Contrast — same model on LM Studio's GGUF backend**
Reasoning still occurs, but it **terminates** and the model produces `content`; the request completes. So this is not "reasoning is too long" in general — MLX specifically fails to close the reasoning segment for this model.
**Secondary note (likely not mlx-engine)**
`chat_template_kwargs: {"enable_thinking": false}` sent in the request body is **not honored on either runtime** via LM Studio's OpenAI-compatible endpoint, so reasoning can't be disabled per-call through the API (the global UI reasoning toggle does work, server-side). This looks like an API-forwarding gap rather than an mlx-engine bug — noting it for context. The llama.cpp CLI honors `--jinja --chat-template-kwargs '{"enable_thinking":false}'` (ggml-org/llama.cpp#21338).
**Impact**
On MLX, Gemma 4 26B-A4B is effectively unusable for any bounded-output task with reasoning enabled — the run never produces content. Current workarounds: disable reasoning via the LM Studio UI toggle, or use the GGUF backend.
**Repro**
1. Load `google/gemma-4-26b-a4b-qat` (MLX), reasoning enabled.
2. POST `/v1/chat/completions` with a prompt that elicits non-trivial reasoning.
3. Observe: `reasoning_content` grows to `max_tokens`, `content` empty, `finish_reason: "length"`.
Contributor guide
Research direction
Start by reproducing the Gemma 4 MLX request described in the issue through the /v1/chat/completions endpoint, then inspect the MLX runtime path that handles reasoning termination and token limits. Compare the result with the same model on the GGUF backend; done means reasoning closes and the request returns non-empty content before exhausting max_tokens.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100