agentscope-ai / agentscope-ai/QwenPaw
[Feature]: Model Provider Layer Optimization — Dynamic Context, Model Sync, Multi-Model Fallback & More
- Dominant language
- Python
- Stars
- 34.9k
- Forks
- 3.1k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 225
Description
## Summary
Comprehensive optimization of the Model Provider layer to reduce manual configuration, improve model discovery, enable multi-model resilience, and unify advanced parameter handling. This umbrella issue tracks 7 interrelated improvements.
## Component(s) Affected
- [x] Core / Backend (app, agents, config, providers, utils, local_models)
- [x] Console (frontend web UI)
- [ ] Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
- [ ] Skills
- [ ] CLI
- [ ] Documentation (website)
- [ ] Tests
- [ ] CI/CD
- [ ] Scripts / Deploy
## Problem / Motivation
The current Model Provider layer has several pain points that increase user friction, especially for new users, and create ongoing maintenance burden:
1. **Context window sizes are mostly hardcoded** — only OpenRouter fetches `context_length` from API; all other 30+ providers rely on a hand-maintained static table or fall back to 128K default. This causes premature context compaction for large-context models (e.g., Gemini 3.1 Pro 1M) or oversized requests for smaller ones.
2. **Model Selector UX is suboptimal** — new users face 50+ models in a flat list with no search, no "recently used", no pin, and no smart recommendations. Experienced users lack quick access to frequently used models.
3. **Free and Pro models share the same management strategy** — Free provider model lists change dynamically (models added/removed), but the system hardcodes them and never auto-updates. Removing a free model has no memory — it reappears on next fetch. Some Free providers (e.g., SiliconFlow) ship with empty model lists (`models=[]`), requiring manual setup.
4. **Each agent supports only one model** — if the active model goes down (API outage, rate limit, deprecation), the agent stops entirely. Sub-agents cannot use cheaper/faster secondary models to save cost.
5. **~800 lines of static model definitions in Python code** — every model update (new release, rename, capability change) requires a code change and release. Capability baselines (`capability_baseline.py`, ~700 lines) duplicate this problem.
6. **Most providers lack `fetch_models()` implementation** — only 4 of 30+ providers (Ollama, LMStudio, OpenRouter, OpenAIProvider base) support dynamic model discovery. Major providers like DashScope, Anthropic, Gemini, Kimi, and DeepSeek do not.
7. **Thinking/Reasoning configuration is model-bound, not agent-bound** — 5 different parameter styles across providers (budget vs. effort vs. tokens), and the config is tied to the model rather than the agent. Switching models can break thinking settings.
## Proposed Solution
### 1. Dynamic Context Window Resolution (P0)
**Current:** Static table (`context_windows.py`) → 128K default.
**Target resolution chain:**
```
User override (highest priority)
↓ not set
Provider API dynamic fetch
↓ API did not return
Built-in static catalog (mainstream models)
↓ no match
128K safe default
```
After model switch, the frontend context usage indicator should immediately reflect the correct window size.
### 2. Frontend Model Selector Redesign (P1)
**Target UX:**
- **Recently Used** section: last 2-3 models, one-click switch
- **Pinned** section: user-pinned favorites
- **Recommended** section (new users): top 6 curated models
- **All Models** (expandable): full list grouped by provider, with PRO/FREE tabs
- Full-text **search** across all available models
### 3. Free/Pro Model Three-Way Sync (P0)
**Free models:**
- Auto-fetch on startup → default visible
- User removal is remembered → not re-added on next fetch
- "Hidden" area to re-add removed models
- "Last synced" timestamp + manual refresh
**Pro models:** Keep current flow (static list + manual add after API key config).
**Sync logic:** `visible = (remote ∪ local_cache) - user_removed + user_pinned`
### 4. Multi-Model Fallback (P0)
- Each agent configurable with **primary model + fallback list**
- Auto-switch on failure with user notification (no workflow interruption)
- Sub-agents can use dedicated secondary models (e.g., main=GPT-5.2, sub=GPT-5-mini)
- Graceful degradation: user sees which model is active, can configure fallback policy, can switch back when primary recovers
### 5. Externalize Model Definitions (P2)
- Move static model lists from Python code to external data files (JSON)
- Enable OTA updates for model catalogs without code release
- Capability baselines auto-fetched from API where possible, manual maintenance as fallback only
### 6. Expand `fetch_models()` Coverage (P1)
- All providers supporting `/v1/models` or equivalent should implement `fetch_models()`
- Flow: configure API key → save → auto-fetch available models → models appear in Selector
- Fetched metadata (context window, multimodal capabilities) auto-applied
- Graceful fallback to built-in static list on fetch failure
### 7. Agent-Level Thinking/Reasoning Budget (P2)
**Current:** Thinking config is per-model, using provider-specific parameters (5 different styles).
**Target:**
- Thinking budget is an **agent-level** setting: Off / Low / Medium / High
- Provider layer auto-maps to provider-specific parameters:
```
Agent: thinking_level = "high"
→ DashScope Qwen: thinking_enable=True, thinking_budget=38400
→ OpenAI: reasoning_effort="high"
→ Anthropic: thinking.budget_tokens=32768
→ Gemini: thinking_config.thinking_budget=32768
```
- Model switch auto-adapts thinking config — no user reconfiguration needed
- Unsupported models: thinking toggle grayed out with explanation
## Priority & Suggested Order
| Priority | Item | User Value |
|:--------:|------|------------|
| P0 | 3. Free/Pro Three-Way Sync | Auto-update + preference memory, critical for new user onboarding |
| P0 | 4. Multi-Model Fallback | Agent resilience when model is unavailable |
| P0 | 1. Dynamic Context Window | Correct model capability utilization |
| P1 | 6. Expand fetch_models | Prerequisite for #3 and #1 |
| P1 | 2. Selector Redesign | Model selection efficiency |
| P2 | 7. Thinking Config Unification | Lower advanced feature barrier |
| P2 | 5. Externalize Model Defs | Long-term maintenance cost |
**Suggested dev order:** 6 → 3 → 1 → 4 → 2 → 7 → 5
## Alternatives Considered
- **Keep static model lists but add more entries:** Does not scale — we have 30+ providers and new models release weekly.
- **Per-provider thinking config UI:** Already exists but creates user confusion on model switch. Agent-level abstraction is cleaner.
- **Single-model-only with manual retry:** Current behavior; unacceptable for production agent workflows.
## Additional Context
- Design document: `docs/designs/model-provider-layer-optimization.md`
- OpenRouter provider already implements dynamic `context_length` and `fetch_models()` — can serve as reference implementation.
- `provider_group` field already exists in provider definitions but only used for frontend grouping, not API key sharing.
## Willing to Contribute
- [x] I am willing to open a PR for this feature (after discussion).
Contributor guide
Assessment
This issue has not been assessed yet.