aaif-goose / aaif-goose/goose

Reasoning models produce empty content: max_tokens shared between reasoning_content and content

Open
#11,142 4 comments 0 reactions 1 assignee Assigned to @jbg View on GitHub
Dominant language
Rust
Stars
54.2k
Forks
6.2k
Avg merge
3d 2h
Merged PRs (30d)
262

Description

## Describe the bug

When using a custom provider with `"reasoning": true`, Goose does not adjust `max_tokens` to account for `reasoning_content` token consumption. The model's thinking tokens count against the same `max_tokens` budget as content tokens. When a reasoning model produces verbose thinking (common with DeepSeek, Nemotron, Qwen3-thinking), the entire budget is consumed by `reasoning_content`, producing **0 content tokens** — the user sees an empty response.

This is a silent failure: no error, no warning, just `finish_reason: "length"` with empty content.

## Root Cause

Goose sends the same `max_tokens` regardless of whether reasoning is enabled:

```
Current behavior:
max_tokens = 4096 (from provider config)
reasoning: true, GOOSE_THINKING_EFFORT: high

Model output:
reasoning_content: ~1200 tokens (thinking)
content: ~0 tokens (ran out of budget)
finish_reason: "length"

User sees: empty response
```

The API contract for OpenAI-compatible servers (llama.cpp, vLLM, Ollama) counts `reasoning_content` + `content` against a single `max_tokens` limit. Goose has the information needed to prevent this (`reasoning: true` flag + `GOOSE_THINKING_EFFORT` setting) but doesn't use it.

## Proposed Solution

Reference implementation from [pi coding agent](https://github.com/earendil-works/pi) (`packages/ai/src/api/simple-options.ts`):

```typescript
const MIN_ANSWER_TOKENS = 1024;

function adjustMaxTokensForThinking(
baseMaxTokens: number,
reasoningLevel: ThinkingLevel,
): { maxTokens: number; thinkingBudget: number } {
const budgets = { minimal: 1024, low: 2048, medium: 8192, high: 16384 };
let thinkingBudget = budgets[reasoningLevel];
let maxTokens = baseMaxTokens + thinkingBudget;

// Guarantee at least MIN_ANSWER_TOKENS for content
if (maxTokens <= thinkingBudget) {
thinkingBudget = Math.max(0, maxTokens - MIN_ANSWER_TOKENS);
}
return { maxTokens, thinkingBudget };
}
```

**Key principle**: `max_tokens = content_budget + thinking_budget` (additive, not shared).

For Goose, when `reasoning: true` and `GOOSE_THINKING_EFFORT != none`:
1. Add thinking budget to `max_tokens` before sending the request
2. Guarantee minimum content tokens (e.g., 1024)
3. This ensures reasoning models always produce visible content

## Evidence

Benchmarked on NVIDIA DGX Spark (GB10, 128 GB) with Nemotron 3.5 Lightning 30B-A3B Q8_0 via llama.cpp:

| Configuration | Quality Score | Content Words | Reasoning Words | Finish Reason |
|---|---|---|---|---|
| `max_tokens: 4096`, reasoning ON | **27%** (13/58) | 0-136 | 950-1400 | `length` ❌ |
| `max_tokens: 4096`, reasoning OFF | **98%** (57/58) | 253-649 | 0 | `stop` ✅ |
| `max_tokens: 6144`, reasoning ON + budget | **97%** (56/58) | 27-495 | 583-1139 | `stop` ✅ |

With the pi-style adjustment (`max_tokens = 4096 + 2048 = 6144`), reasoning AND content both fit — 97% quality with reasoning enabled.

This affects **all local reasoning models**: DeepSeek-V4-Flash, Nemotron 3.5, Qwen3.6 (thinking mode), and any future reasoning model served via llama.cpp/vLLM/Ollama.

## To Reproduce

1. Configure a custom provider pointing to a reasoning model (e.g., llama.cpp serving Nemotron or DeepSeek):
```json
{
"name": "custom_spark_nemotron",
"engine": "openai",
"base_url": "http://localhost:8084/v1",
"models": [{
"name": "nemotron-35-lightning",
"max_tokens": 4096,
"reasoning": true
}]
}
```

2. Set `GOOSE_THINKING_EFFORT: high` in config

3. Ask Goose to write code: *"Write a Python function with type hints and a docstring"*

4. Observe: empty response (all tokens consumed by `reasoning_content`)

## Expected behavior

Goose should adjust `max_tokens` when `reasoning: true` to accommodate both `reasoning_content` and `content`, guaranteeing the user always sees a response.

## Environment

- **OS & Arch:** macOS 14.x arm64 (client), Ubuntu 24.04 aarch64 (server)
- **Interface:** CLI
- **Version:** 1.45.0
- **Provider & Model:** Custom OpenAI-compatible → llama.cpp → Nemotron 3.5 Lightning 30B-A3B / DeepSeek-V4-Flash
- **Extensions enabled:** N/A

## Additional context

- **llama.cpp** (the most common local inference server) counts `reasoning_content` against `max_tokens` — this is the default API behavior
- **`reasoning_effort: none`** is the only workaround (disables reasoning entirely), but this loses the quality benefits of thinking
- **`--reasoning-budget N`** server flag can cap thinking, but requires manual coordination between client `max_tokens` and server config
- The `reasoning: true` flag in provider config already signals that Goose should account for reasoning tokens — it just doesn't act on it
- Related issues: #10821 (reasoning consistency), #11092 (MCP sampling for reasoning-first responses)

Full benchmark methodology and results: [docs/20-quality-benchmark.md](https://github.com/vincenzopalazzo/local-ai/blob/main/docs/20-quality-benchmark.md)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.