HarperFast / HarperFast/harper

ollama backend: generate() returns empty content for thinking models (response tokens land in 'thinking', never read)

Open
#1,805 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

`models.generate()` through the Ollama backend returns an **empty `content`** when the configured model is a thinking/reasoning model (e.g. `qwen3-next`): Ollama routes all output tokens into the response's `thinking` field (thinking defaults on for capable models), and the backend reads only `data.response` (`components/ollama/index.ts` — `rawContent = endpoint === '/api/chat' ? data.message?.content : data.response`). The caller gets `{ content: "", finishReason: "stop" }` with no signal that anything went wrong.

With `responseFormat: { schema }` this is especially surprising: the request asks for schema-constrained output and receives an empty string that fails any downstream parse.

## Repro (ollama 0.30.x, qwen3-next:80b, direct request matching what the backend sends)

`POST /api/generate` with `{ model: "qwen3-next:80b", prompt: "", stream: false, format: , options: { temperature: 0.2, num_predict: 2000 } }`:

| `think` | `response` length | `thinking` length | JSON parse of `response` |
|---|---|---|---|
| absent (default) | **0** | 344 | fails (empty) |
| `false` | 320 | 0 | valid, schema-conformant |
| `true` | 0 | 330 | fails (empty) |

Same prompt against a non-thinking model (`qwen3-coder-next`) succeeds with `think` absent. So the failure is purely the thinking default interacting with `/api/generate`'s response routing — the backend never sets `think`, and it never reads `data.thinking`.

## Suggested direction

Any of these would fix the empty-content trap, in rough order of preference:

1. **Default `think: false` in `buildGenerateRequest`** for non-streaming `generate` — the facade's contract is "return the answer in `content`"; reasoning tokens the caller can't see are spent budget with no output. (An explicit opt-in could come later via a `GenerateOpts` field.)
2. Expose `think` (or a provider-options passthrough) on `GenerateOpts` so callers can control it.
3. At minimum: when `data.response` is empty but `data.thinking` is non-empty, throw a descriptive backend error instead of returning empty content — silent empties are the worst failure mode.

Happy to provide the full repro script. Found while integrating `models.generate()` downstream (tpsdev-ai/flair#707): any deployment that points the ollama backend at a thinking model gets empty generations with no diagnostic.

Contributor guide

Open the contributing guide

Research direction

Start in components/ollama/index.ts, tracing buildGenerateRequest and the /api/generate handling used by models.generate. Reproduce the empty-content case with a thinking model and compare data.response with data.thinking. Done means the backend no longer silently returns empty content for this case, using an agreed request or diagnostic behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, ollama
Domain
ai, api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.