apache / apache/tooling-llmao

enable_thinking does not survive the proxy

Open
#31 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

chat_template_kwargs: {"enable_thinking": true} produces zero reasoning characters through llm.apache.org on all three models, including Gemma where reasoning is off by default and the kwarg is the only way to turn it on.

Direct to the box, the same kwarg works -- a 27B request with enable_thinking unset returned 50 tokens entirely of reasoning and content: null.

So the field is being dropped or rewritten in transit. Likely LiteLLM filtering unknown keys, possibly drop_params: true in litellm.yaml.

Why it matters: it is the only control callers have over reasoning. Without it a latency-sensitive caller cannot turn it off and a caller wanting deliberation cannot turn it on. The catalog documents thinks_by_default on the assumption the override works.

To check: direct vs proxy with the same body, then whether drop_params: true is responsible.

Contributor guide

No contributing guide indexed for this repository

Research direction

Compare direct and proxy requests using the same body, then inspect whether the proxy drops chat_template_kwargs, especially through the LiteLLM configuration mentioned in litellm.yaml. Check whether drop_params: true causes the behavior. Done means enable_thinking reaches all three models and toggles reasoning consistently through llm.apache.org.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.