agentscope-ai / agentscope-ai/agentscope-java

[Feature]: Multi-level model fallback chain (fallbackModels) with failure classification and per-candidate cooldown

Đang mở
#2,863 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Java
Star
5.6k
Fork
1.3k
Merge trung bình
4 ngày 12 giờ
Pull request đã merge (30 ngày)
77

Mô tả

## Motivation

Model failover today is single-level: `ReActAgent.Builder.fallbackModel(...)` accepts exactly one backup model, and `modelForCall()` switches to it on the first error signal of the primary stream. In production, a provider outage or a revoked API key commonly takes down more than one model of the same vendor, so a single fallback is frequently exhausted too.

The concrete gaps this feature addresses:

1. **Single-layer fallback** — the fallback itself has no further escape hatch; when both primary and fallback fail the request is lost even if a third model would work.
2. **No failure classification** — a request-side error (400/422), which will fail identically on every candidate, still switches and burns the fallback budget.
3. **No cooldown** — a persistently broken model is hammered on every request until the agent is rebuilt; there is no "skip the known-bad model for a while" state.
4. **No transparency** — callers cannot observe which candidate actually served the request (`getModelName()` reports the primary even after a switch) or that a cooldown is active.

## Proposal

Add a pluggable, opt-in multi-level fallback chain as a public `Model` wrapper:

- **`FallbackChainModel`** (new public class in `io.agentscope.core.model`): wraps an ordered list `[primary, fb1, fb2, ...]`; on a failure that switching can recover from (429 / 5xx / timeout / network / 401 / 403) it transparently moves to the next candidate; on a request-side failure (400/422 and other 4xx) it fails fast without consuming the chain.
- **Per-candidate cooldown**: after a failure a candidate is skipped for a cooldown window (default 30s), then automatically becomes eligible again — lazy recovery verified by real traffic, no scheduler or background threads (keeps the core library lightweight).
- **Mid-stream failures** (first chunk already delivered) are deliberately not retried on a fallback — switching mid-response can duplicate delivered content; the failure is recorded (cooldown applied) and propagated as-is.
- **Capability delegation**: `getModelName()` / `supportsNativeStructuredOutput()` / `getContextWindowSize()` report the currently active candidate.
- **Builder switch** `ReActAgent.Builder.fallbackModels(Model...)`: when configured, `modelForCall()` wires the chain; when unset, behaviour is byte-for-byte the existing single-fallback path (zero default-change, backward compatible). Users can also bypass the builder and wire `model(new FallbackChainModel(primary, fallbacks))` directly.
- **Agent-scoped shared cooldown table**: cooldown survives across the many reasoning rounds of a loop and successive requests (each `modelForCall()` creates a fresh wrapper but they share the agent's cooldown table), while the active-model reference stays per-call for concurrency safety.

## API shape (draft)

```java
ReActAgent.builder()
.model(primary)
.fallbackModels(
List.of(
ModelRegistry.resolve("dashscope:qwen-max"),
ModelRegistry.resolve("openai:gpt-4o-mini")))
.build();

// or directly:
Model m = new FallbackChainModel(primary, fallbacks); // default 30s cooldown
Model m2 = new FallbackChainModel(primary, fallbacks, Duration.ofSeconds(10));
```

## Compatibility

- Legacy `fallbackModel(Model/String)` and `maxRetries(int)` untouched.
- `ModelConfig` record untouched (zero breaking change).
- No new dependencies, no threads, no external state.
- Designed so a future explicit health-probe (issue follow-up) can drive recovery instead of lazy traffic verification without API changes.

## Tests

- `FallbackChainModelTest` (12 cases): primary success, switch on switchable error, full-chain exhaustion, 401/403 switch, timeout/network switch, 400 fail-fast, mid-stream no-switch + cooldown, cooldown skip + lazy recovery, shared-table persistence across wrappers, null/empty handling.
- `ReActAgentFallbackChainTest` (4 integration cases): agent falls back on 503, primary-only default, fail-fast does not consume chain, direct wrapper wiring via `.model(...)`.

## Related

- Paves the way for #2136 (typed retry/fallback attempt events) by making the active candidate observable through `getModelName()`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.