agentscope-ai / agentscope-ai/agentscope-java
[Feature]: Multi-level model fallback chain (fallbackModels) with failure classification and per-candidate cooldown
- Dominant language
- Java
- Stars
- 5.6k
- Forks
- 1.3k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 77
Description
## Motivation
Model failover today is single-level: `ReActAgent.Builder.fallbackModel(...)` accepts exactly one backup model, and `modelForCall()` switches to it on the first error signal of the primary stream. In production, a provider outage or a revoked API key commonly takes down more than one model of the same vendor, so a single fallback is frequently exhausted too.
The concrete gaps this feature addresses:
1. **Single-layer fallback** — the fallback itself has no further escape hatch; when both primary and fallback fail the request is lost even if a third model would work.
2. **No failure classification** — a request-side error (400/422), which will fail identically on every candidate, still switches and burns the fallback budget.
3. **No cooldown** — a persistently broken model is hammered on every request until the agent is rebuilt; there is no "skip the known-bad model for a while" state.
4. **No transparency** — callers cannot observe which candidate actually served the request (`getModelName()` reports the primary even after a switch) or that a cooldown is active.
## Proposal
Add a pluggable, opt-in multi-level fallback chain as a public `Model` wrapper:
- **`FallbackChainModel`** (new public class in `io.agentscope.core.model`): wraps an ordered list `[primary, fb1, fb2, ...]`; on a failure that switching can recover from (429 / 5xx / timeout / network / 401 / 403) it transparently moves to the next candidate; on a request-side failure (400/422 and other 4xx) it fails fast without consuming the chain.
- **Per-candidate cooldown**: after a failure a candidate is skipped for a cooldown window (default 30s), then automatically becomes eligible again — lazy recovery verified by real traffic, no scheduler or background threads (keeps the core library lightweight).
- **Mid-stream failures** (first chunk already delivered) are deliberately not retried on a fallback — switching mid-response can duplicate delivered content; the failure is recorded (cooldown applied) and propagated as-is.
- **Capability delegation**: `getModelName()` / `supportsNativeStructuredOutput()` / `getContextWindowSize()` report the currently active candidate.
- **Builder switch** `ReActAgent.Builder.fallbackModels(Model...)`: when configured, `modelForCall()` wires the chain; when unset, behaviour is byte-for-byte the existing single-fallback path (zero default-change, backward compatible). Users can also bypass the builder and wire `model(new FallbackChainModel(primary, fallbacks))` directly.
- **Agent-scoped shared cooldown table**: cooldown survives across the many reasoning rounds of a loop and successive requests (each `modelForCall()` creates a fresh wrapper but they share the agent's cooldown table), while the active-model reference stays per-call for concurrency safety.
## API shape (draft)
```java
ReActAgent.builder()
.model(primary)
.fallbackModels(
List.of(
ModelRegistry.resolve("dashscope:qwen-max"),
ModelRegistry.resolve("openai:gpt-4o-mini")))
.build();
// or directly:
Model m = new FallbackChainModel(primary, fallbacks); // default 30s cooldown
Model m2 = new FallbackChainModel(primary, fallbacks, Duration.ofSeconds(10));
```
## Compatibility
- Legacy `fallbackModel(Model/String)` and `maxRetries(int)` untouched.
- `ModelConfig` record untouched (zero breaking change).
- No new dependencies, no threads, no external state.
- Designed so a future explicit health-probe (issue follow-up) can drive recovery instead of lazy traffic verification without API changes.
## Tests
- `FallbackChainModelTest` (12 cases): primary success, switch on switchable error, full-chain exhaustion, 401/403 switch, timeout/network switch, 400 fail-fast, mid-stream no-switch + cooldown, cooldown skip + lazy recovery, shared-table persistence across wrappers, null/empty handling.
- `ReActAgentFallbackChainTest` (4 integration cases): agent falls back on 503, primary-only default, fail-fast does not consume chain, direct wrapper wiring via `.model(...)`.
## Related
- Paves the way for #2136 (typed retry/fallback attempt events) by making the active candidate observable through `getModelName()`.
Contributor guide
Assessment
This issue has not been assessed yet.