enable_thinking does not survive the proxy
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
chat_template_kwargs: {"enable_thinking": true} produces zero reasoning characters through llm.apache.org on all three models, including Gemma where reasoning is off by default and the kwarg is the only way to turn it on.
Direct to the box, the same kwarg works -- a 27B request with enable_thinking unset returned 50 tokens entirely of reasoning and content: null.
So the field is being dropped or rewritten in transit. Likely LiteLLM filtering unknown keys, possibly drop_params: true in litellm.yaml.
Why it matters: it is the only control callers have over reasoning. Without it a latency-sensitive caller cannot turn it off and a caller wanting deliberation cannot turn it on. The catalog documents thinks_by_default on the assumption the override works.
To check: direct vs proxy with the same body, then whether drop_params: true is responsible.
Contributor guide
No contributing guide indexed for this repository
Research direction
Compare direct and proxy requests using the same body, then inspect whether the proxy drops chat_template_kwargs, especially through the LiteLLM configuration mentioned in litellm.yaml. Check whether drop_params: true causes the behavior. Done means enable_thinking reaches all three models and toggles reasoning consistently through llm.apache.org.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100