agentscope-ai / agentscope-ai/agentscope-java

[Feature]: Multi-level model fallback chain (fallbackModels) with failure classification and per-candidate cooldown

Open
#2,863 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
5.6k
Forks
1.3k
Avg merge
4d 12h
Merged PRs (30d)
77

Description

## Motivation

Model failover today is single-level: `ReActAgent.Builder.fallbackModel(...)` accepts exactly one backup model, and `modelForCall()` switches to it on the first error signal of the primary stream. In production, a provider outage or a revoked API key commonly takes down more than one model of the same vendor, so a single fallback is frequently exhausted too.

The concrete gaps this feature addresses:

1. **Single-layer fallback** — the fallback itself has no further escape hatch; when both primary and fallback fail the request is lost even if a third model would work.
2. **No failure classification** — a request-side error (400/422), which will fail identically on every candidate, still switches and burns the fallback budget.
3. **No cooldown** — a persistently broken model is hammered on every request until the agent is rebuilt; there is no "skip the known-bad model for a while" state.
4. **No transparency** — callers cannot observe which candidate actually served the request (`getModelName()` reports the primary even after a switch) or that a cooldown is active.

## Proposal

Add a pluggable, opt-in multi-level fallback chain as a public `Model` wrapper:

- **`FallbackChainModel`** (new public class in `io.agentscope.core.model`): wraps an ordered list `[primary, fb1, fb2, ...]`; on a failure that switching can recover from (429 / 5xx / timeout / network / 401 / 403) it transparently moves to the next candidate; on a request-side failure (400/422 and other 4xx) it fails fast without consuming the chain.
- **Per-candidate cooldown**: after a failure a candidate is skipped for a cooldown window (default 30s), then automatically becomes eligible again — lazy recovery verified by real traffic, no scheduler or background threads (keeps the core library lightweight).
- **Mid-stream failures** (first chunk already delivered) are deliberately not retried on a fallback — switching mid-response can duplicate delivered content; the failure is recorded (cooldown applied) and propagated as-is.
- **Capability delegation**: `getModelName()` / `supportsNativeStructuredOutput()` / `getContextWindowSize()` report the currently active candidate.
- **Builder switch** `ReActAgent.Builder.fallbackModels(Model...)`: when configured, `modelForCall()` wires the chain; when unset, behaviour is byte-for-byte the existing single-fallback path (zero default-change, backward compatible). Users can also bypass the builder and wire `model(new FallbackChainModel(primary, fallbacks))` directly.
- **Agent-scoped shared cooldown table**: cooldown survives across the many reasoning rounds of a loop and successive requests (each `modelForCall()` creates a fresh wrapper but they share the agent's cooldown table), while the active-model reference stays per-call for concurrency safety.

## API shape (draft)

```java
ReActAgent.builder()
.model(primary)
.fallbackModels(
List.of(
ModelRegistry.resolve("dashscope:qwen-max"),
ModelRegistry.resolve("openai:gpt-4o-mini")))
.build();

// or directly:
Model m = new FallbackChainModel(primary, fallbacks); // default 30s cooldown
Model m2 = new FallbackChainModel(primary, fallbacks, Duration.ofSeconds(10));
```

## Compatibility

- Legacy `fallbackModel(Model/String)` and `maxRetries(int)` untouched.
- `ModelConfig` record untouched (zero breaking change).
- No new dependencies, no threads, no external state.
- Designed so a future explicit health-probe (issue follow-up) can drive recovery instead of lazy traffic verification without API changes.

## Tests

- `FallbackChainModelTest` (12 cases): primary success, switch on switchable error, full-chain exhaustion, 401/403 switch, timeout/network switch, 400 fail-fast, mid-stream no-switch + cooldown, cooldown skip + lazy recovery, shared-table persistence across wrappers, null/empty handling.
- `ReActAgentFallbackChainTest` (4 integration cases): agent falls back on 503, primary-only default, fail-fast does not consume chain, direct wrapper wiring via `.model(...)`.

## Related

- Paves the way for #2136 (typed retry/fallback attempt events) by making the active candidate observable through `getModelName()`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.