Reasoning models produce empty content: max_tokens shared between reasoning_content and content
- Langage dominant
- Rust
- Étoiles
- 54.2k
- Forks
- 6.2k
- Merge moyen
- 3 j 4 h
- PR mergées (30 j)
- 240
Description
## Describe the bug
When using a custom provider with `"reasoning": true`, Goose does not adjust `max_tokens` to account for `reasoning_content` token consumption. The model's thinking tokens count against the same `max_tokens` budget as content tokens. When a reasoning model produces verbose thinking (common with DeepSeek, Nemotron, Qwen3-thinking), the entire budget is consumed by `reasoning_content`, producing **0 content tokens** — the user sees an empty response.
This is a silent failure: no error, no warning, just `finish_reason: "length"` with empty content.
## Root Cause
Goose sends the same `max_tokens` regardless of whether reasoning is enabled:
```
Current behavior:
max_tokens = 4096 (from provider config)
reasoning: true, GOOSE_THINKING_EFFORT: high
Model output:
reasoning_content: ~1200 tokens (thinking)
content: ~0 tokens (ran out of budget)
finish_reason: "length"
User sees: empty response
```
The API contract for OpenAI-compatible servers (llama.cpp, vLLM, Ollama) counts `reasoning_content` + `content` against a single `max_tokens` limit. Goose has the information needed to prevent this (`reasoning: true` flag + `GOOSE_THINKING_EFFORT` setting) but doesn't use it.
## Proposed Solution
Reference implementation from [pi coding agent](https://github.com/earendil-works/pi) (`packages/ai/src/api/simple-options.ts`):
```typescript
const MIN_ANSWER_TOKENS = 1024;
function adjustMaxTokensForThinking(
baseMaxTokens: number,
reasoningLevel: ThinkingLevel,
): { maxTokens: number; thinkingBudget: number } {
const budgets = { minimal: 1024, low: 2048, medium: 8192, high: 16384 };
let thinkingBudget = budgets[reasoningLevel];
let maxTokens = baseMaxTokens + thinkingBudget;
// Guarantee at least MIN_ANSWER_TOKENS for content
if (maxTokens <= thinkingBudget) {
thinkingBudget = Math.max(0, maxTokens - MIN_ANSWER_TOKENS);
}
return { maxTokens, thinkingBudget };
}
```
**Key principle**: `max_tokens = content_budget + thinking_budget` (additive, not shared).
For Goose, when `reasoning: true` and `GOOSE_THINKING_EFFORT != none`:
1. Add thinking budget to `max_tokens` before sending the request
2. Guarantee minimum content tokens (e.g., 1024)
3. This ensures reasoning models always produce visible content
## Evidence
Benchmarked on NVIDIA DGX Spark (GB10, 128 GB) with Nemotron 3.5 Lightning 30B-A3B Q8_0 via llama.cpp:
| Configuration | Quality Score | Content Words | Reasoning Words | Finish Reason |
|---|---|---|---|---|
| `max_tokens: 4096`, reasoning ON | **27%** (13/58) | 0-136 | 950-1400 | `length` ❌ |
| `max_tokens: 4096`, reasoning OFF | **98%** (57/58) | 253-649 | 0 | `stop` ✅ |
| `max_tokens: 6144`, reasoning ON + budget | **97%** (56/58) | 27-495 | 583-1139 | `stop` ✅ |
With the pi-style adjustment (`max_tokens = 4096 + 2048 = 6144`), reasoning AND content both fit — 97% quality with reasoning enabled.
This affects **all local reasoning models**: DeepSeek-V4-Flash, Nemotron 3.5, Qwen3.6 (thinking mode), and any future reasoning model served via llama.cpp/vLLM/Ollama.
## To Reproduce
1. Configure a custom provider pointing to a reasoning model (e.g., llama.cpp serving Nemotron or DeepSeek):
```json
{
"name": "custom_spark_nemotron",
"engine": "openai",
"base_url": "http://localhost:8084/v1",
"models": [{
"name": "nemotron-35-lightning",
"max_tokens": 4096,
"reasoning": true
}]
}
```
2. Set `GOOSE_THINKING_EFFORT: high` in config
3. Ask Goose to write code: *"Write a Python function with type hints and a docstring"*
4. Observe: empty response (all tokens consumed by `reasoning_content`)
## Expected behavior
Goose should adjust `max_tokens` when `reasoning: true` to accommodate both `reasoning_content` and `content`, guaranteeing the user always sees a response.
## Environment
- **OS & Arch:** macOS 14.x arm64 (client), Ubuntu 24.04 aarch64 (server)
- **Interface:** CLI
- **Version:** 1.45.0
- **Provider & Model:** Custom OpenAI-compatible → llama.cpp → Nemotron 3.5 Lightning 30B-A3B / DeepSeek-V4-Flash
- **Extensions enabled:** N/A
## Additional context
- **llama.cpp** (the most common local inference server) counts `reasoning_content` against `max_tokens` — this is the default API behavior
- **`reasoning_effort: none`** is the only workaround (disables reasoning entirely), but this loses the quality benefits of thinking
- **`--reasoning-budget N`** server flag can cap thinking, but requires manual coordination between client `max_tokens` and server config
- The `reasoning: true` flag in provider config already signals that Goose should account for reasoning tokens — it just doesn't act on it
- Related issues: #10821 (reasoning consistency), #11092 (MCP sampling for reasoning-first responses)
Full benchmark methodology and results: [docs/20-quality-benchmark.md](https://github.com/vincenzopalazzo/local-ai/blob/main/docs/20-quality-benchmark.md)
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Le problème concerne la manière dont Goose gère le budget de tokens pour les modèles de raisonnement. Examine la logique de configuration du fournisseur où `max_tokens` est défini, en particulier lorsque le flag `reasoning` vaut true. L’algorithme d’ajustement est décrit dans l’issue ; implémente-le pour ajouter un budget de réflexion. Teste avec un serveur local de modèle de raisonnement (comme llama.cpp) afin de vérifier que reasoning_content et content sont tous deux produits.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- ollama, rust
- Domaine
- ai, ai-infra-agents, backend-api-design
- Type d'issue
- Bug
- Difficulté
- 3/5
- Temps estimé
- 1-2 jours
- Activité
- Active
- Clarté
- Clairement spécifiée
- Accessibilité débutants
- 55/100