code-yeongyu / code-yeongyu/senpi
catalog: toggle-only reasoning models advertise a fake graded thinking ladder (missing thinkingLevelMap)
- Dominant language
- TypeScript
- Stars
- 429
- Forks
- 98
- Avg merge
- 5h 3m
- Merged PRs (30d)
- 526
Description
## Summary
A bounded catalog-accuracy audit (run as part of the per-model reasoning work, plan `fast-reasoning-effort-commands`) found that **62 builtin catalog models advertise a graded reasoning ladder they do not actually support**.
`getSupportedThinkingLevels()` (packages/ai/src/models.ts:917-928) returns the FULL extended ladder for any `reasoning: true` model that has no `thinkingLevelMap`. Toggle-only models (thinkingFormat `zai`, `qwen`, `qwen-chat-template`, `string-thinking`, `deepseek`, `together`) therefore expose fake `minimal/low/medium/xhigh` levels, and the new `classifyReasoningCapability()` helper labels them `graded` instead of `on-off`/`always-on`.
## Why this is filed instead of fixed inline
The fix must go through the generator (`packages/ai/scripts/generate-models.ts`) + `npm run hydrate:model-data` — never hand-edits to generated files. A scratch hydrate on 2026-08-16 showed regeneration currently drags in **unrelated upstream catalog drift**:
- new models: `glm-5.3`, `deepseek-v4-pro-0813`, `@cf/deepseek-v4-flash-0731`, `gemini-3.7-flash`, `Qwen2.5-Coder-32B-Instruct`, `meta/muse-glimmer-30b`
- removal: `cloudflare-ai-gateway: claude-3-5-haiku`
- limit bump: `baseten` 1000000 -> 1048576
Landing that churn inside a behavioral PR would make it unreviewable, so the audit was deliberately split out.
## Recommended sequence
1. A standalone **catalog refresh** PR that lands the upstream drift alone (pure `hydrate:model-data` output, reviewed as data).
2. Then a generator-source PR adding the missing `thinkingLevelMap` entries (sections: zai, moonshot, xiaomi, qwen-token-plan, alibaba-token-plan, opencode*) plus `packages/coding-agent/test/catalog-thinking-map.test.ts` asserting each corrected model's capability.
## Models flagged `needs-map` (62)
| alibaba-token-plan | `deepseek-v3.2` | deepseek | none | models.dev alibaba-token-plan: reasoning_options=[{type:'toggle'}]; repo lists deepseek-v3.2 in QWEN_TOKEN_PLAN_REASONING_EFFORT_UNSUPPORTED_MODEL_IDS | needs-map (toggle) |
| alibaba-token-plan | `kimi-k2.5` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| alibaba-token-plan | `kimi-k2.6` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| alibaba-token-plan | `kimi-k2.7-code` | deepseek | none | Moonshot doc: kimi-k2.7-code thinking always-on (moonshotapi). models.dev claims toggle on DashScope/alibaba endpoints | needs-map (toggle on resellers; moonshot-direct is always-on and already mapped) |
| alibaba-token-plan | `qwen3.6-flash` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| alibaba-token-plan | `qwen3.6-plus` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| alibaba-token-plan | `qwen3.7-max` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| alibaba-token-plan | `qwen3.7-plus` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| alibaba-token-plan | `qwen3.8-max` | qwen | none | QwenCloud thinking.md: qwen3.8-max reasoning_effort options low/medium/xhigh (default xhigh) | needs-map (graded low/medium/xhigh) |
| alibaba-token-plan | `qwen3.8-max-preview` | qwen | none | QwenCloud thinking.md: qwen3.8-max reasoning_effort options low/medium/xhigh (default xhigh) | needs-map (graded low/medium/xhigh) |
| moonshotai | `kimi-k2-thinking` | deepseek | none | Moonshot models.md: 'kimi-k2 series discontinued on May 25, 2026'; no current per-parameter doc; family-consistent toggle | needs-map (toggle; undocumented-primary — model deprecated) |
| moonshotai | `kimi-k2-thinking-turbo` | deepseek | none | Moonshot models.md: 'kimi-k2 series discontinued on May 25, 2026'; family-consistent toggle | needs-map (toggle; undocumented-primary — model deprecated) |
| moonshotai | `kimi-k2.5` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| moonshotai | `kimi-k2.6` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| moonshotai-cn | `kimi-k2-thinking` | deepseek | none | Moonshot models.md: 'kimi-k2 series discontinued on May 25, 2026'; no current per-parameter doc; family-consistent toggle | needs-map (toggle; undocumented-primary — model deprecated) |
| moonshotai-cn | `kimi-k2-thinking-turbo` | deepseek | none | Moonshot models.md: 'kimi-k2 series discontinued on May 25, 2026'; family-consistent toggle | needs-map (toggle; undocumented-primary — model deprecated) |
| moonshotai-cn | `kimi-k2.5` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| moonshotai-cn | `kimi-k2.6` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| opencode | `kimi-k2.6` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| opencode-go | `qwen3.6-plus` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan | `MiniMax-M2.5` | qwen | none | models.dev reasoning_options=[] (unclassified); repo already maps together MiniMax family as toggle {minimal:null,low:null,medium:null} | needs-map (toggle; unclassified-secondary) |
| qwen-token-plan | `deepseek-v3.2` | qwen | none | models.dev alibaba-token-plan: reasoning_options=[{type:'toggle'}]; repo lists deepseek-v3.2 in QWEN_TOKEN_PLAN_REASONING_EFFORT_UNSUPPORTED_MODEL_IDS | needs-map (toggle) |
| qwen-token-plan | `kimi-k2.5` | qwen | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| qwen-token-plan | `kimi-k2.6` | qwen | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| qwen-token-plan | `kimi-k2.7-code` | qwen | none | Moonshot doc: kimi-k2.7-code thinking always-on (moonshotapi). models.dev claims toggle on DashScope/alibaba endpoints | needs-map (toggle on resellers; moonshot-direct is always-on and already mapped) |
| qwen-token-plan | `qwen3.6-flash` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan | `qwen3.6-plus` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan | `qwen3.7-max` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan | `qwen3.7-plus` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan-cn | `MiniMax-M2.5` | qwen | none | models.dev reasoning_options=[] (unclassified); repo already maps together MiniMax family as toggle {minimal:null,low:null,medium:null} | needs-map (toggle; unclassified-secondary) |
| qwen-token-plan-cn | `deepseek-v3.2` | qwen | none | models.dev alibaba-token-plan: reasoning_options=[{type:'toggle'}]; repo lists deepseek-v3.2 in QWEN_TOKEN_PLAN_REASONING_EFFORT_UNSUPPORTED_MODEL_IDS | needs-map (toggle) |
| qwen-token-plan-cn | `kimi-k2.5` | qwen | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| qwen-token-plan-cn | `kimi-k2.6` | qwen | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| qwen-token-plan-cn | `kimi-k2.7-code` | qwen | none | Moonshot doc: kimi-k2.7-code thinking always-on (moonshotapi). models.dev claims toggle on DashScope/alibaba endpoints | needs-map (toggle on resellers; moonshot-direct is always-on and already mapped) |
| qwen-token-plan-cn | `qwen3.6-flash` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan-cn | `qwen3.6-plus` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan-cn | `qwen3.7-max` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan-cn | `qwen3.7-plus` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan-individual | `qwen3.6-flash` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan-individual | `qwen3.7-max` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan-individual | `qwen3.7-plus` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| xiaomi | `mimo-v2-flash` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi | `mimo-v2-omni` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi | `mimo-v2-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi | `mimo-v2.5` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled'; no reasoning_effort field in doc | needs-map (toggle) |
| xiaomi | `mimo-v2.5-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled' | needs-map (toggle) |
| xiaomi | `mimo-v2.5-pro-ultraspeed` | deepseek | none | Xiaomi OpenAI-compat API doc (v2.5 family toggle); ultraspeed variant doc'd via marketing page; no graded effort | needs-map (toggle) |
| xiaomi-token-plan-ams | `mimo-v2-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi-token-plan-ams | `mimo-v2.5` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled'; no reasoning_effort field in doc | needs-map (toggle) |
| xiaomi-token-plan-ams | `mimo-v2.5-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled' | needs-map (toggle) |
| xiaomi-token-plan-cn | `mimo-v2-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi-token-plan-cn | `mimo-v2.5` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled'; no reasoning_effort field in doc | needs-map (toggle) |
| xiaomi-token-plan-cn | `mimo-v2.5-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled' | needs-map (toggle) |
| xiaomi-token-plan-sgp | `mimo-v2-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi-token-plan-sgp | `mimo-v2.5` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled'; no reasoning_effort field in doc | needs-map (toggle) |
| xiaomi-token-plan-sgp | `mimo-v2.5-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled' | needs-map (toggle) |
| zai | `glm-4.7` | zai | none | Z.AI API ref: reasoning_effort is 'Only supported by GLM-5.2'; GLM-4.7 accepts thinking.type enabled/disabled only | needs-map (toggle) |
| zai | `glm-5-turbo` | zai | none | Z.AI API ref: reasoning_effort 'Only supported by GLM-5.2'; thinking.type enabled/disabled | needs-map (toggle) |
| zai | `glm-5.2-highspeed` | zai | none | models.dev zai-coding-plan: reasoning_options=[{type:'effort',values:['high','max']}] (same as glm-5.2); id absent from Z.AI public API model enum | needs-map (graded high/max; undocumented-primary) |
| zai-coding-cn | `glm-4.7` | zai | none | Z.AI API ref: reasoning_effort is 'Only supported by GLM-5.2'; GLM-4.7 accepts thinking.type enabled/disabled only | needs-map (toggle) |
| zai-coding-cn | `glm-5-turbo` | zai | none | Z.AI API ref: reasoning_effort 'Only supported by GLM-5.2'; thinking.type enabled/disabled | needs-map (toggle) |
| zai-coding-cn | `glm-5.2-highspeed` | zai | none | models.dev zai-coding-plan: reasoning_options=[{type:'effort',values:['high','max']}] (same as glm-5.2); id absent from Z.AI public API model enum | needs-map (graded high/max; undocumented-primary) |
## Additional findings
- **Xiaomi transport contradiction:** compat flag `supportsDisabledThinking: false` contradicts Xiaomi's own API docs (`thinking.type: "disabled"` is accepted). A toggle map exposing `off` would advertise a level the current transport cannot actually disable. Transport-side fix, separate concern.
- **`zai/glm-5.2`:** map omits `xhigh` although Z.AI docs map `xhigh -> max`. Outside the "no map" filter of this audit, but wrong.
- **Deprecated entries:** `kimi-k2-thinking`, `kimi-k2-thinking-turbo` (Moonshot discontinued the kimi-k2 series 2026-05-25) and `MiMo-V2` are better deleted than mapped.
- **Weak-source rows:** `MiniMax-M2.5` (models.dev `reasoning_options=[]`) and `glm-5.2-highspeed` (absent from Z.AI's public model enum) rest on secondary sources; confirm before mapping.
Provider documentation quotes with URLs are recorded in the audit artifact produced by that run.
_Filed by the `fast-reasoning-effort-commands` work plan; the behavioral PR deliberately leaves `packages/ai` byte-identical (84 generated artifacts verified unchanged)._
Contributor guide
Research direction
Start with packages/ai/src/models.ts:917-928 and the classifyReasoningCapability() behavior, then inspect the generator sections named in packages/ai/scripts/generate-models.ts. Run npm run hydrate:model-data while checking the catalog refresh boundaries, and use packages/coding-agent/test/catalog-thinking-map.test.ts to verify the corrected models expose only their documented reasoning capabilities without unrelated catalog churn.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100