Reasoning models produce empty content: max_tokens shared between reasoning_content and content
- Dominant language
- Rust
- Stars
- 54.2k
- Forks
- 6.2k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 262
Description
## Describe the bug
When using a custom provider with `"reasoning": true`, Goose does not adjust `max_tokens` to account for `reasoning_content` token consumption. The model's thinking tokens count against the same `max_tokens` budget as content tokens. When a reasoning model produces verbose thinking (common with DeepSeek, Nemotron, Qwen3-thinking), the entire budget is consumed by `reasoning_content`, producing **0 content tokens** — the user sees an empty response.
This is a silent failure: no error, no warning, just `finish_reason: "length"` with empty content.
## Root Cause
Goose sends the same `max_tokens` regardless of whether reasoning is enabled:
```
Current behavior:
max_tokens = 4096 (from provider config)
reasoning: true, GOOSE_THINKING_EFFORT: high
Model output:
reasoning_content: ~1200 tokens (thinking)
content: ~0 tokens (ran out of budget)
finish_reason: "length"
User sees: empty response
```
The API contract for OpenAI-compatible servers (llama.cpp, vLLM, Ollama) counts `reasoning_content` + `content` against a single `max_tokens` limit. Goose has the information needed to prevent this (`reasoning: true` flag + `GOOSE_THINKING_EFFORT` setting) but doesn't use it.
## Proposed Solution
Reference implementation from [pi coding agent](https://github.com/earendil-works/pi) (`packages/ai/src/api/simple-options.ts`):
```typescript
const MIN_ANSWER_TOKENS = 1024;
function adjustMaxTokensForThinking(
baseMaxTokens: number,
reasoningLevel: ThinkingLevel,
): { maxTokens: number; thinkingBudget: number } {
const budgets = { minimal: 1024, low: 2048, medium: 8192, high: 16384 };
let thinkingBudget = budgets[reasoningLevel];
let maxTokens = baseMaxTokens + thinkingBudget;
// Guarantee at least MIN_ANSWER_TOKENS for content
if (maxTokens <= thinkingBudget) {
thinkingBudget = Math.max(0, maxTokens - MIN_ANSWER_TOKENS);
}
return { maxTokens, thinkingBudget };
}
```
**Key principle**: `max_tokens = content_budget + thinking_budget` (additive, not shared).
For Goose, when `reasoning: true` and `GOOSE_THINKING_EFFORT != none`:
1. Add thinking budget to `max_tokens` before sending the request
2. Guarantee minimum content tokens (e.g., 1024)
3. This ensures reasoning models always produce visible content
## Evidence
Benchmarked on NVIDIA DGX Spark (GB10, 128 GB) with Nemotron 3.5 Lightning 30B-A3B Q8_0 via llama.cpp:
| Configuration | Quality Score | Content Words | Reasoning Words | Finish Reason |
|---|---|---|---|---|
| `max_tokens: 4096`, reasoning ON | **27%** (13/58) | 0-136 | 950-1400 | `length` ❌ |
| `max_tokens: 4096`, reasoning OFF | **98%** (57/58) | 253-649 | 0 | `stop` ✅ |
| `max_tokens: 6144`, reasoning ON + budget | **97%** (56/58) | 27-495 | 583-1139 | `stop` ✅ |
With the pi-style adjustment (`max_tokens = 4096 + 2048 = 6144`), reasoning AND content both fit — 97% quality with reasoning enabled.
This affects **all local reasoning models**: DeepSeek-V4-Flash, Nemotron 3.5, Qwen3.6 (thinking mode), and any future reasoning model served via llama.cpp/vLLM/Ollama.
## To Reproduce
1. Configure a custom provider pointing to a reasoning model (e.g., llama.cpp serving Nemotron or DeepSeek):
```json
{
"name": "custom_spark_nemotron",
"engine": "openai",
"base_url": "http://localhost:8084/v1",
"models": [{
"name": "nemotron-35-lightning",
"max_tokens": 4096,
"reasoning": true
}]
}
```
2. Set `GOOSE_THINKING_EFFORT: high` in config
3. Ask Goose to write code: *"Write a Python function with type hints and a docstring"*
4. Observe: empty response (all tokens consumed by `reasoning_content`)
## Expected behavior
Goose should adjust `max_tokens` when `reasoning: true` to accommodate both `reasoning_content` and `content`, guaranteeing the user always sees a response.
## Environment
- **OS & Arch:** macOS 14.x arm64 (client), Ubuntu 24.04 aarch64 (server)
- **Interface:** CLI
- **Version:** 1.45.0
- **Provider & Model:** Custom OpenAI-compatible → llama.cpp → Nemotron 3.5 Lightning 30B-A3B / DeepSeek-V4-Flash
- **Extensions enabled:** N/A
## Additional context
- **llama.cpp** (the most common local inference server) counts `reasoning_content` against `max_tokens` — this is the default API behavior
- **`reasoning_effort: none`** is the only workaround (disables reasoning entirely), but this loses the quality benefits of thinking
- **`--reasoning-budget N`** server flag can cap thinking, but requires manual coordination between client `max_tokens` and server config
- The `reasoning: true` flag in provider config already signals that Goose should account for reasoning tokens — it just doesn't act on it
- Related issues: #10821 (reasoning consistency), #11092 (MCP sampling for reasoning-first responses)
Full benchmark methodology and results: [docs/20-quality-benchmark.md](https://github.com/vincenzopalazzo/local-ai/blob/main/docs/20-quality-benchmark.md)
Contributor guide
Assessment
This issue has not been assessed yet.