docker / docker/docker-agent

thinking_budget: none is silently ignored on OpenAI-compatible endpoints, and provider_opts cannot pass the vendor off switch

Open
#4,309 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/providers area/providers/docker-model-runner area/providers/openai
Dominant language
Go
Stars
3.3k
Forks
462
Avg merge
1d 10h
Merged PRs (30d)
273

Description

Open-weight reasoning models (Qwen3, DeepSeek, GLM) think by default, and every OpenAI-compatible server or vendor exposes a request field to turn it off: llama.cpp, vLLM, SGLang and mlx_lm accept chat_template_kwargs: {"enable_thinking": false}; llama.cpp, vLLM, SGLang, Ollama, Groq and Cerebras accept reasoning_effort: "none"; DeepSeek uses thinking: {"type": "disabled"}. docker agent can send none of them: thinking_budget is only acted on for OpenAI reasoning model names (modelinfo.UsesReasoningEffort), a disabled budget is normalised to nil in applyModelDefaults before any client sees it (which also leaves the documented DMR reasoning-budget: 0 path dead), and provider_opts forwarding is limited to the sampling allowlist. The model therefore reasons on every call and max_tokens can be spent entirely on reasoning.

Measured with v1.139.0 on macOS against mlx_lm.server 0.31.3 serving mlx-community/Qwen3.6-35B-A3B-8bit:

models:
  local:
    provider: openai
    model: mlx-community/Qwen3.6-35B-A3B-8bit
    base_url: http://localhost:8080/v1
    temperature: 0

docker agent run --exec --json agent.yaml - with a 7 kB prompt took 80–200 s per call and emitted 4.5k–13k agent_choice_reasoning events; the same prompt with chat_template_kwargs sent directly returns in about 3 s. thinking_budget: none, provider_opts: {chat_template_kwargs: {enable_thinking: false}} and a /no_think prefix all changed nothing (209–231 reasoning events on a three-word probe); a proxy confirms the body carries only model, messages, stream, temperature. With max_tokens: 2048 the whole budget went to reasoning: zero agent_choice events, exit 0, a warning event, no error. provider: dmr with runtime_flags: ["--reasoning-budget", "0"] on qwen3.6:35b-a3b-q8_0 still produced 218 reasoning events (whether the flag reached llama.cpp is unconfirmed; a failed _configure is only logged at debug level).

Proposed:

  1. provider_opts.extra_body (object) merged verbatim into the chat-completions body for the openai and dmr clients, on every provider, as the general escape hatch (extra_body: {reasoning_effort: none} on groq, extra_body: {thinking: {type: disabled}} on deepseek, ...).
  2. thinking_budget: none / 0 on a user-supplied base_url (not Azure/ChatGPT, model name not an OpenAI one) sends chat_template_kwargs: {"enable_thinking": false} on Chat Completions with the existing 256-token max_tokens floor; the dmr client sends the same per request alongside llamacpp.reasoning-budget, which also covers its MLX and SGLang engines.
  3. dmr: internal no-thinking clones (MCP sampling) no longer call the model-wide _configure. Model Runner replaces a model's stored configuration on every call, so a sampling call already wiped the agent's reasoning-budget (verified: 4096 → {} in docker model configure show); with the disabled budget preserved it would set it to 0 instead. Title and compaction clones already skipped it.

Follow-up, not in scope: a per-alias off switch in the alias registry so thinking_budget: none works on built-in aliases such as ollama (which honours reasoning_effort: none) without extra_body.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at applyModelDefaults and trace how thinking_budget and provider_opts reach the OpenAI and DMR request builders, then inspect the Model Runner path for MCP sampling clones. Verify the request bodies against the mlx_lm and DMR scenarios described, including preserved disabled budgets and extra_body forwarding; done means the proposed no-thinking controls work without breaking existing reasoning-model behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai, api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.