HarperFast / HarperFast/harper
ollama backend: generate() returns empty content for thinking models (response tokens land in 'thinking', never read)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
`models.generate()` through the Ollama backend returns an **empty `content`** when the configured model is a thinking/reasoning model (e.g. `qwen3-next`): Ollama routes all output tokens into the response's `thinking` field (thinking defaults on for capable models), and the backend reads only `data.response` (`components/ollama/index.ts` — `rawContent = endpoint === '/api/chat' ? data.message?.content : data.response`). The caller gets `{ content: "", finishReason: "stop" }` with no signal that anything went wrong.
With `responseFormat: { schema }` this is especially surprising: the request asks for schema-constrained output and receives an empty string that fails any downstream parse.
## Repro (ollama 0.30.x, qwen3-next:80b, direct request matching what the backend sends)
`POST /api/generate` with `{ model: "qwen3-next:80b", prompt: "", stream: false, format: , options: { temperature: 0.2, num_predict: 2000 } }`:
| `think` | `response` length | `thinking` length | JSON parse of `response` |
|---|---|---|---|
| absent (default) | **0** | 344 | fails (empty) |
| `false` | 320 | 0 | valid, schema-conformant |
| `true` | 0 | 330 | fails (empty) |
Same prompt against a non-thinking model (`qwen3-coder-next`) succeeds with `think` absent. So the failure is purely the thinking default interacting with `/api/generate`'s response routing — the backend never sets `think`, and it never reads `data.thinking`.
## Suggested direction
Any of these would fix the empty-content trap, in rough order of preference:
1. **Default `think: false` in `buildGenerateRequest`** for non-streaming `generate` — the facade's contract is "return the answer in `content`"; reasoning tokens the caller can't see are spent budget with no output. (An explicit opt-in could come later via a `GenerateOpts` field.)
2. Expose `think` (or a provider-options passthrough) on `GenerateOpts` so callers can control it.
3. At minimum: when `data.response` is empty but `data.thinking` is non-empty, throw a descriptive backend error instead of returning empty content — silent empties are the worst failure mode.
Happy to provide the full repro script. Found while integrating `models.generate()` downstream (tpsdev-ai/flair#707): any deployment that points the ollama backend at a thinking model gets empty generations with no diagnostic.
Contributor guide
Research direction
Start in components/ollama/index.ts, tracing buildGenerateRequest and the /api/generate handling used by models.generate. Reproduce the empty-content case with a thinking model and compare data.response with data.thinking. Done means the backend no longer silently returns empty content for this case, using an agreed request or diagnostic behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, ollama
- Domain
- ai, api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100