code-yeongyu / code-yeongyu/senpi

catalog: toggle-only reasoning models advertise a fake graded thinking ladder (missing thinkingLevelMap)

Open
#891 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
429
Forks
98
Avg merge
5h 3m
Merged PRs (30d)
526

Description

## Summary

A bounded catalog-accuracy audit (run as part of the per-model reasoning work, plan `fast-reasoning-effort-commands`) found that **62 builtin catalog models advertise a graded reasoning ladder they do not actually support**.

`getSupportedThinkingLevels()` (packages/ai/src/models.ts:917-928) returns the FULL extended ladder for any `reasoning: true` model that has no `thinkingLevelMap`. Toggle-only models (thinkingFormat `zai`, `qwen`, `qwen-chat-template`, `string-thinking`, `deepseek`, `together`) therefore expose fake `minimal/low/medium/xhigh` levels, and the new `classifyReasoningCapability()` helper labels them `graded` instead of `on-off`/`always-on`.

## Why this is filed instead of fixed inline

The fix must go through the generator (`packages/ai/scripts/generate-models.ts`) + `npm run hydrate:model-data` — never hand-edits to generated files. A scratch hydrate on 2026-08-16 showed regeneration currently drags in **unrelated upstream catalog drift**:

- new models: `glm-5.3`, `deepseek-v4-pro-0813`, `@cf/deepseek-v4-flash-0731`, `gemini-3.7-flash`, `Qwen2.5-Coder-32B-Instruct`, `meta/muse-glimmer-30b`
- removal: `cloudflare-ai-gateway: claude-3-5-haiku`
- limit bump: `baseten` 1000000 -> 1048576

Landing that churn inside a behavioral PR would make it unreviewable, so the audit was deliberately split out.

## Recommended sequence

1. A standalone **catalog refresh** PR that lands the upstream drift alone (pure `hydrate:model-data` output, reviewed as data).
2. Then a generator-source PR adding the missing `thinkingLevelMap` entries (sections: zai, moonshot, xiaomi, qwen-token-plan, alibaba-token-plan, opencode*) plus `packages/coding-agent/test/catalog-thinking-map.test.ts` asserting each corrected model's capability.

## Models flagged `needs-map` (62)

| alibaba-token-plan | `deepseek-v3.2` | deepseek | none | models.dev alibaba-token-plan: reasoning_options=[{type:'toggle'}]; repo lists deepseek-v3.2 in QWEN_TOKEN_PLAN_REASONING_EFFORT_UNSUPPORTED_MODEL_IDS | needs-map (toggle) |
| alibaba-token-plan | `kimi-k2.5` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| alibaba-token-plan | `kimi-k2.6` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| alibaba-token-plan | `kimi-k2.7-code` | deepseek | none | Moonshot doc: kimi-k2.7-code thinking always-on (moonshotapi). models.dev claims toggle on DashScope/alibaba endpoints | needs-map (toggle on resellers; moonshot-direct is always-on and already mapped) |
| alibaba-token-plan | `qwen3.6-flash` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| alibaba-token-plan | `qwen3.6-plus` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| alibaba-token-plan | `qwen3.7-max` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| alibaba-token-plan | `qwen3.7-plus` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| alibaba-token-plan | `qwen3.8-max` | qwen | none | QwenCloud thinking.md: qwen3.8-max reasoning_effort options low/medium/xhigh (default xhigh) | needs-map (graded low/medium/xhigh) |
| alibaba-token-plan | `qwen3.8-max-preview` | qwen | none | QwenCloud thinking.md: qwen3.8-max reasoning_effort options low/medium/xhigh (default xhigh) | needs-map (graded low/medium/xhigh) |
| moonshotai | `kimi-k2-thinking` | deepseek | none | Moonshot models.md: 'kimi-k2 series discontinued on May 25, 2026'; no current per-parameter doc; family-consistent toggle | needs-map (toggle; undocumented-primary — model deprecated) |
| moonshotai | `kimi-k2-thinking-turbo` | deepseek | none | Moonshot models.md: 'kimi-k2 series discontinued on May 25, 2026'; family-consistent toggle | needs-map (toggle; undocumented-primary — model deprecated) |
| moonshotai | `kimi-k2.5` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| moonshotai | `kimi-k2.6` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| moonshotai-cn | `kimi-k2-thinking` | deepseek | none | Moonshot models.md: 'kimi-k2 series discontinued on May 25, 2026'; no current per-parameter doc; family-consistent toggle | needs-map (toggle; undocumented-primary — model deprecated) |
| moonshotai-cn | `kimi-k2-thinking-turbo` | deepseek | none | Moonshot models.md: 'kimi-k2 series discontinued on May 25, 2026'; family-consistent toggle | needs-map (toggle; undocumented-primary — model deprecated) |
| moonshotai-cn | `kimi-k2.5` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| moonshotai-cn | `kimi-k2.6` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| opencode | `kimi-k2.6` | deepseek | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| opencode-go | `qwen3.6-plus` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan | `MiniMax-M2.5` | qwen | none | models.dev reasoning_options=[] (unclassified); repo already maps together MiniMax family as toggle {minimal:null,low:null,medium:null} | needs-map (toggle; unclassified-secondary) |
| qwen-token-plan | `deepseek-v3.2` | qwen | none | models.dev alibaba-token-plan: reasoning_options=[{type:'toggle'}]; repo lists deepseek-v3.2 in QWEN_TOKEN_PLAN_REASONING_EFFORT_UNSUPPORTED_MODEL_IDS | needs-map (toggle) |
| qwen-token-plan | `kimi-k2.5` | qwen | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| qwen-token-plan | `kimi-k2.6` | qwen | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| qwen-token-plan | `kimi-k2.7-code` | qwen | none | Moonshot doc: kimi-k2.7-code thinking always-on (moonshotapi). models.dev claims toggle on DashScope/alibaba endpoints | needs-map (toggle on resellers; moonshot-direct is always-on and already mapped) |
| qwen-token-plan | `qwen3.6-flash` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan | `qwen3.6-plus` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan | `qwen3.7-max` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan | `qwen3.7-plus` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan-cn | `MiniMax-M2.5` | qwen | none | models.dev reasoning_options=[] (unclassified); repo already maps together MiniMax family as toggle {minimal:null,low:null,medium:null} | needs-map (toggle; unclassified-secondary) |
| qwen-token-plan-cn | `deepseek-v3.2` | qwen | none | models.dev alibaba-token-plan: reasoning_options=[{type:'toggle'}]; repo lists deepseek-v3.2 in QWEN_TOKEN_PLAN_REASONING_EFFORT_UNSUPPORTED_MODEL_IDS | needs-map (toggle) |
| qwen-token-plan-cn | `kimi-k2.5` | qwen | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| qwen-token-plan-cn | `kimi-k2.6` | qwen | none | Moonshot Thinking Models doc: thinking.type enabled/disabled; reasoning_effort 'Not supported' | needs-map (toggle) |
| qwen-token-plan-cn | `kimi-k2.7-code` | qwen | none | Moonshot doc: kimi-k2.7-code thinking always-on (moonshotapi). models.dev claims toggle on DashScope/alibaba endpoints | needs-map (toggle on resellers; moonshot-direct is always-on and already mapped) |
| qwen-token-plan-cn | `qwen3.6-flash` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan-cn | `qwen3.6-plus` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan-cn | `qwen3.7-max` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan-cn | `qwen3.7-plus` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan-individual | `qwen3.6-flash` | qwen | none | QwenCloud thinking.md: 'Hybrid: toggle thinking on or off per request with enable_thinking'; reasoning_effort not documented for 3.6 (only 3.8-max) | needs-map (toggle) |
| qwen-token-plan-individual | `qwen3.7-max` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| qwen-token-plan-individual | `qwen3.7-plus` | qwen | none | QwenCloud thinking.md: enable_thinking hybrid; thinking_budget supported but reasoning_effort not documented for 3.7 | needs-map (toggle) |
| xiaomi | `mimo-v2-flash` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi | `mimo-v2-omni` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi | `mimo-v2-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi | `mimo-v2.5` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled'; no reasoning_effort field in doc | needs-map (toggle) |
| xiaomi | `mimo-v2.5-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled' | needs-map (toggle) |
| xiaomi | `mimo-v2.5-pro-ultraspeed` | deepseek | none | Xiaomi OpenAI-compat API doc (v2.5 family toggle); ultraspeed variant doc'd via marketing page; no graded effort | needs-map (toggle) |
| xiaomi-token-plan-ams | `mimo-v2-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi-token-plan-ams | `mimo-v2.5` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled'; no reasoning_effort field in doc | needs-map (toggle) |
| xiaomi-token-plan-ams | `mimo-v2.5-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled' | needs-map (toggle) |
| xiaomi-token-plan-cn | `mimo-v2-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi-token-plan-cn | `mimo-v2.5` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled'; no reasoning_effort field in doc | needs-map (toggle) |
| xiaomi-token-plan-cn | `mimo-v2.5-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled' | needs-map (toggle) |
| xiaomi-token-plan-sgp | `mimo-v2-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: thinking.type enabled/disabled (named for v2.5); MiMo-V2 series deprecated June 30; no reasoning_effort anywhere in doc | needs-map (toggle; series deprecated) |
| xiaomi-token-plan-sgp | `mimo-v2.5` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled'; no reasoning_effort field in doc | needs-map (toggle) |
| xiaomi-token-plan-sgp | `mimo-v2.5-pro` | deepseek | none | Xiaomi OpenAI-compat API doc: 'mimo-v2.5-pro, mimo-v2.5: default enabled; Available options: enabled, disabled' | needs-map (toggle) |
| zai | `glm-4.7` | zai | none | Z.AI API ref: reasoning_effort is 'Only supported by GLM-5.2'; GLM-4.7 accepts thinking.type enabled/disabled only | needs-map (toggle) |
| zai | `glm-5-turbo` | zai | none | Z.AI API ref: reasoning_effort 'Only supported by GLM-5.2'; thinking.type enabled/disabled | needs-map (toggle) |
| zai | `glm-5.2-highspeed` | zai | none | models.dev zai-coding-plan: reasoning_options=[{type:'effort',values:['high','max']}] (same as glm-5.2); id absent from Z.AI public API model enum | needs-map (graded high/max; undocumented-primary) |
| zai-coding-cn | `glm-4.7` | zai | none | Z.AI API ref: reasoning_effort is 'Only supported by GLM-5.2'; GLM-4.7 accepts thinking.type enabled/disabled only | needs-map (toggle) |
| zai-coding-cn | `glm-5-turbo` | zai | none | Z.AI API ref: reasoning_effort 'Only supported by GLM-5.2'; thinking.type enabled/disabled | needs-map (toggle) |
| zai-coding-cn | `glm-5.2-highspeed` | zai | none | models.dev zai-coding-plan: reasoning_options=[{type:'effort',values:['high','max']}] (same as glm-5.2); id absent from Z.AI public API model enum | needs-map (graded high/max; undocumented-primary) |

## Additional findings

- **Xiaomi transport contradiction:** compat flag `supportsDisabledThinking: false` contradicts Xiaomi's own API docs (`thinking.type: "disabled"` is accepted). A toggle map exposing `off` would advertise a level the current transport cannot actually disable. Transport-side fix, separate concern.
- **`zai/glm-5.2`:** map omits `xhigh` although Z.AI docs map `xhigh -> max`. Outside the "no map" filter of this audit, but wrong.
- **Deprecated entries:** `kimi-k2-thinking`, `kimi-k2-thinking-turbo` (Moonshot discontinued the kimi-k2 series 2026-05-25) and `MiMo-V2` are better deleted than mapped.
- **Weak-source rows:** `MiniMax-M2.5` (models.dev `reasoning_options=[]`) and `glm-5.2-highspeed` (absent from Z.AI's public model enum) rest on secondary sources; confirm before mapping.

Provider documentation quotes with URLs are recorded in the audit artifact produced by that run.

_Filed by the `fast-reasoning-effort-commands` work plan; the behavioral PR deliberately leaves `packages/ai` byte-identical (84 generated artifacts verified unchanged)._

Contributor guide

Open the contributing guide

Research direction

Start with packages/ai/src/models.ts:917-928 and the classifyReasoningCapability() behavior, then inspect the generator sections named in packages/ai/scripts/generate-models.ts. Run npm run hydrate:model-data while checking the catalog refresh boundaries, and use packages/coding-agent/test/catalog-thinking-map.test.ts to verify the corrected models expose only their documented reasoning capabilities without unrelated catalog churn.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.