lmstudio-ai / lmstudio-ai/mlx-engine

Gemma 4 (26B-A4B) reasoning never terminates on MLX — fills max_tokens with reasoning_content, empty content (GGUF terminates fine)

Open
#337 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
133
Avg merge
21h 6m
Merged PRs (30d)
1

Description

**Summary**

On the MLX runtime, Gemma 4 26B-A4B reasoning **never terminates**: the model emits `reasoning_content` until it hits `max_tokens` and returns empty `content` (`finish_reason: "length"`). The *same* model on LM Studio's llama.cpp/GGUF backend reasons but **terminates normally** and returns content — so this non-termination appears MLX-engine-specific.

**Environment**
- LM Studio MLX engine `0.4.16+2`
- Model: `google/gemma-4-26b-a4b-qat` (MLX, Q4 QAT)
- macOS, Apple Silicon (M4 Pro)

**Observed (MLX)**
```json
"choices": [{ "message": { "content": "", "reasoning_content": "...~16k tokens..." }, "finish_reason": "length" }],
"usage": { "completion_tokens": 15999, "completion_tokens_details": { "reasoning_tokens": 15997 } }
```
`content` is empty; the entire budget is consumed by reasoning that never closes.

**Contrast — same model on LM Studio's GGUF backend**
Reasoning still occurs, but it **terminates** and the model produces `content`; the request completes. So this is not "reasoning is too long" in general — MLX specifically fails to close the reasoning segment for this model.

**Secondary note (likely not mlx-engine)**
`chat_template_kwargs: {"enable_thinking": false}` sent in the request body is **not honored on either runtime** via LM Studio's OpenAI-compatible endpoint, so reasoning can't be disabled per-call through the API (the global UI reasoning toggle does work, server-side). This looks like an API-forwarding gap rather than an mlx-engine bug — noting it for context. The llama.cpp CLI honors `--jinja --chat-template-kwargs '{"enable_thinking":false}'` (ggml-org/llama.cpp#21338).

**Impact**

On MLX, Gemma 4 26B-A4B is effectively unusable for any bounded-output task with reasoning enabled — the run never produces content. Current workarounds: disable reasoning via the LM Studio UI toggle, or use the GGUF backend.

**Repro**
1. Load `google/gemma-4-26b-a4b-qat` (MLX), reasoning enabled.
2. POST `/v1/chat/completions` with a prompt that elicits non-trivial reasoning.
3. Observe: `reasoning_content` grows to `max_tokens`, `content` empty, `finish_reason: "length"`.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Gemma 4 MLX request described in the issue through the /v1/chat/completions endpoint, then inspect the MLX runtime path that handles reasoning termination and token limits. Compare the result with the same model on the GGUF backend; done means reasoning closes and the request returns non-empty content before exhausting max_tokens.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.