agentscope-ai / agentscope-ai/QwenPaw

[Feature]: Model Provider Layer Optimization — Dynamic Context, Model Sync, Multi-Model Fallback & More

Open
#6,167 1 comment 1 reaction 1 assignee Claimed by @wangfei010313 View on GitHub
enhancement
Dominant language
Python
Stars
34.9k
Forks
3.1k
Avg merge
1d 15h
Merged PRs (30d)
225

Description

## Summary

Comprehensive optimization of the Model Provider layer to reduce manual configuration, improve model discovery, enable multi-model resilience, and unify advanced parameter handling. This umbrella issue tracks 7 interrelated improvements.

## Component(s) Affected

- [x] Core / Backend (app, agents, config, providers, utils, local_models)
- [x] Console (frontend web UI)
- [ ] Channels (DingTalk, Feishu, QQ, Discord, iMessage, etc.)
- [ ] Skills
- [ ] CLI
- [ ] Documentation (website)
- [ ] Tests
- [ ] CI/CD
- [ ] Scripts / Deploy

## Problem / Motivation

The current Model Provider layer has several pain points that increase user friction, especially for new users, and create ongoing maintenance burden:

1. **Context window sizes are mostly hardcoded** — only OpenRouter fetches `context_length` from API; all other 30+ providers rely on a hand-maintained static table or fall back to 128K default. This causes premature context compaction for large-context models (e.g., Gemini 3.1 Pro 1M) or oversized requests for smaller ones.

2. **Model Selector UX is suboptimal** — new users face 50+ models in a flat list with no search, no "recently used", no pin, and no smart recommendations. Experienced users lack quick access to frequently used models.

3. **Free and Pro models share the same management strategy** — Free provider model lists change dynamically (models added/removed), but the system hardcodes them and never auto-updates. Removing a free model has no memory — it reappears on next fetch. Some Free providers (e.g., SiliconFlow) ship with empty model lists (`models=[]`), requiring manual setup.

4. **Each agent supports only one model** — if the active model goes down (API outage, rate limit, deprecation), the agent stops entirely. Sub-agents cannot use cheaper/faster secondary models to save cost.

5. **~800 lines of static model definitions in Python code** — every model update (new release, rename, capability change) requires a code change and release. Capability baselines (`capability_baseline.py`, ~700 lines) duplicate this problem.

6. **Most providers lack `fetch_models()` implementation** — only 4 of 30+ providers (Ollama, LMStudio, OpenRouter, OpenAIProvider base) support dynamic model discovery. Major providers like DashScope, Anthropic, Gemini, Kimi, and DeepSeek do not.

7. **Thinking/Reasoning configuration is model-bound, not agent-bound** — 5 different parameter styles across providers (budget vs. effort vs. tokens), and the config is tied to the model rather than the agent. Switching models can break thinking settings.

## Proposed Solution

### 1. Dynamic Context Window Resolution (P0)

**Current:** Static table (`context_windows.py`) → 128K default.

**Target resolution chain:**
```
User override (highest priority)
↓ not set
Provider API dynamic fetch
↓ API did not return
Built-in static catalog (mainstream models)
↓ no match
128K safe default
```

After model switch, the frontend context usage indicator should immediately reflect the correct window size.

### 2. Frontend Model Selector Redesign (P1)

**Target UX:**
- **Recently Used** section: last 2-3 models, one-click switch
- **Pinned** section: user-pinned favorites
- **Recommended** section (new users): top 6 curated models
- **All Models** (expandable): full list grouped by provider, with PRO/FREE tabs
- Full-text **search** across all available models

### 3. Free/Pro Model Three-Way Sync (P0)

**Free models:**
- Auto-fetch on startup → default visible
- User removal is remembered → not re-added on next fetch
- "Hidden" area to re-add removed models
- "Last synced" timestamp + manual refresh

**Pro models:** Keep current flow (static list + manual add after API key config).

**Sync logic:** `visible = (remote ∪ local_cache) - user_removed + user_pinned`

### 4. Multi-Model Fallback (P0)

- Each agent configurable with **primary model + fallback list**
- Auto-switch on failure with user notification (no workflow interruption)
- Sub-agents can use dedicated secondary models (e.g., main=GPT-5.2, sub=GPT-5-mini)
- Graceful degradation: user sees which model is active, can configure fallback policy, can switch back when primary recovers

### 5. Externalize Model Definitions (P2)

- Move static model lists from Python code to external data files (JSON)
- Enable OTA updates for model catalogs without code release
- Capability baselines auto-fetched from API where possible, manual maintenance as fallback only

### 6. Expand `fetch_models()` Coverage (P1)

- All providers supporting `/v1/models` or equivalent should implement `fetch_models()`
- Flow: configure API key → save → auto-fetch available models → models appear in Selector
- Fetched metadata (context window, multimodal capabilities) auto-applied
- Graceful fallback to built-in static list on fetch failure

### 7. Agent-Level Thinking/Reasoning Budget (P2)

**Current:** Thinking config is per-model, using provider-specific parameters (5 different styles).

**Target:**
- Thinking budget is an **agent-level** setting: Off / Low / Medium / High
- Provider layer auto-maps to provider-specific parameters:
```
Agent: thinking_level = "high"
→ DashScope Qwen: thinking_enable=True, thinking_budget=38400
→ OpenAI: reasoning_effort="high"
→ Anthropic: thinking.budget_tokens=32768
→ Gemini: thinking_config.thinking_budget=32768
```
- Model switch auto-adapts thinking config — no user reconfiguration needed
- Unsupported models: thinking toggle grayed out with explanation

## Priority & Suggested Order

| Priority | Item | User Value |
|:--------:|------|------------|
| P0 | 3. Free/Pro Three-Way Sync | Auto-update + preference memory, critical for new user onboarding |
| P0 | 4. Multi-Model Fallback | Agent resilience when model is unavailable |
| P0 | 1. Dynamic Context Window | Correct model capability utilization |
| P1 | 6. Expand fetch_models | Prerequisite for #3 and #1 |
| P1 | 2. Selector Redesign | Model selection efficiency |
| P2 | 7. Thinking Config Unification | Lower advanced feature barrier |
| P2 | 5. Externalize Model Defs | Long-term maintenance cost |

**Suggested dev order:** 6 → 3 → 1 → 4 → 2 → 7 → 5

## Alternatives Considered

- **Keep static model lists but add more entries:** Does not scale — we have 30+ providers and new models release weekly.
- **Per-provider thinking config UI:** Already exists but creates user confusion on model switch. Agent-level abstraction is cleaner.
- **Single-model-only with manual retry:** Current behavior; unacceptable for production agent workflows.

## Additional Context

- Design document: `docs/designs/model-provider-layer-optimization.md`
- OpenRouter provider already implements dynamic `context_length` and `fetch_models()` — can serve as reference implementation.
- `provider_group` field already exists in provider definitions but only used for frontend grouping, not API key sharing.

## Willing to Contribute

- [x] I am willing to open a PR for this feature (after discussion).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.