feat(catgrad-llm): support thinking switches and structured input/outputs
- Dominant language
- Rust
- Stars
- 34
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
some models have a "thinking" mode that can be toggled on/off- qwen
some support high/med/low- gpt-oss
openai/anthropic conversation types have a 'thinking' content type- check if/where this is supported as an input to the template and render it correctly
do we need to parse model output to construct the typed 'thinking' outputs that the proprietary apis return? do other servers (e.g llamacpp) do this? if so, same logic as #454 applies- if we did not request thinking but model outputs thinking.. what do we do? render as text output?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how templates accept OpenAI/Anthropic conversation content and how model responses are represented, then compare other servers such as llamacpp and the decisions in #454. Done means the thinking switch levels, structured thinking inputs, and typed or text outputs have defined behavior across the supported cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100