[API Proposal]: Add CascadingChatClient for response-quality routing

### Background and motivation

> **Update (2026-08-22):** This proposal was revised based on the discussion below. It now proposes a separate `CascadingChatClient` instead of extending `FailoverChatClient`.

The experimental routing APIs (`[Experimental("MEAI001")]`) introduced `RoutingChatClient`, `SemanticRoutingChatClient`, `FailoverChatClient`, and `OrderedFailoverChatClient`.

These types support selecting a client for a request and selecting another client after an eligible provider failure. What is still missing is an ordered routing mode that can continue after a request succeeds but the application does not accept the response quality.

That is the model-cascading pattern described by approaches such as [FrugalGPT](https://arxiv.org/abs/2305.05176): start with a less expensive model, evaluate the completed response, and use a stronger model only when necessary.

This proposal does not define how quality is measured. That decision can remain in the application or evaluation layer and may use a predicate, a confidence score, `Microsoft.Extensions.AI.Evaluation`, or another application-specific policy.

Cascading and failover have different completion semantics. Failover is driven by provider failures and, for streaming, whether output has already been committed. Cascading is driven by a quality decision over a successful, completed `ChatResponse`. Keeping them as separate routing clients gives each behavior a smaller and clearer contract.

The two behaviors can still be combined through composition. Each cascade rung is an `IChatClient`, so a rung can itself be a resilient `FailoverChatClient`. The inner client handles provider failures within a tier, while the outer `CascadingChatClient` handles quality escalation between tiers.

A working proof of concept is available here:

https://github.com/WittBen/meai-quality-cascade

### API Proposal

```csharp
namespace Microsoft.Extensions.AI;

[Experimental("MEAI001")]
public abstract class CascadingChatClient : RoutingChatClient
{
protected CascadingChatClient(
IReadOnlyList clients,
bool leaveOpen = false);

protected abstract ValueTask IsResponseAcceptedAsync(
RoutingContext context,
IChatClient client,
ChatResponse response,
CancellationToken cancellationToken);
}
```

No changes are proposed for `RoutingContext`, `FailoverChatClient`, or `FailoverChatClientAttempt`. In particular, this revision no longer proposes `RoutingContext.AttemptNumber`, `FailoverChatClientAttempt.Response`, or `FailoverChatClient.ShouldSelectAgainAsync`.

#### Behavioral contract

The constructor receives a non-empty ordered list of chat clients. The list is snapshotted and must not contain `null` entries.

Each request creates its own `RoutingContext` and starts with the first configured client. After a client returns a successful `ChatResponse`:

- If another client remains, `IsResponseAcceptedAsync` is called.
- Returning `true` returns the response immediately.
- Returning `false` discards the response and invokes the next client.
- The final client's response is returned without another quality decision.

The final-response rule ensures that the finite client list always terminates. An application that requires a hard quality floor can validate the returned response outside the cascade.

The quality hook receives the request's `RoutingContext`, the client that produced the response, the completed `ChatResponse`, and the request cancellation token. This allows an application to apply request- or model-specific policy without introducing a judging abstraction into MEAI.

A provider exception, evaluation exception, or cancellation is propagated to the caller and does not cause quality escalation. Provider failover can be added by using a `FailoverChatClient` as an individual cascade rung.

The current rung is request-scoped and safe for concurrent requests. The implementation can follow the existing `OrderedFailoverChatClient` pattern and use a `ConcurrentDictionary`. Retained state is removed after completion, provider failure, evaluation failure, cancellation, or disposal.

`CascadingChatClient` does not provide native streaming. `GetStreamingResponseAsync` uses the non-streaming cascade path and converts the accepted response with `ChatResponse.ToChatResponseUpdates()`. The returned updates therefore do not provide native time-to-first-token latency.

The cascade owns and disposes its configured clients by default. Passing `leaveOpen: true` leaves their lifetime with the caller.

### API Usage

A small concrete implementation can keep the quality policy in application code:

```csharp
internal sealed class QualityCascadeChatClient : CascadingChatClient
{
private readonly Func<
RoutingContext,
IChatClient,
ChatResponse,
CancellationToken,
ValueTask> _isResponseAccepted;

public QualityCascadeChatClient(
IReadOnlyList clients,
Func<
RoutingContext,
IChatClient,
ChatResponse,
CancellationToken,
ValueTask> isResponseAccepted,
bool leaveOpen = false)
: base(clients, leaveOpen)
{
_isResponseAccepted = isResponseAccepted;
}

protected override ValueTask IsResponseAcceptedAsync(
RoutingContext context,
IChatClient client,
ChatResponse response,
CancellationToken cancellationToken)
=> _isResponseAccepted(context, client, response, cancellationToken);
}
```

The application supplies the ordered clients and decides what an acceptable response means:

```csharp
using IChatClient client = new QualityCascadeChatClient(
clients: [cheapClient, strongClient],
isResponseAccepted: static (context, selectedClient, response, cancellationToken) =>
{
cancellationToken.ThrowIfCancellationRequested();

double quality =
response.AdditionalProperties?.TryGetValue("quality", out object? value) == true
? Convert.ToDouble(value)
: 0;

return new ValueTask(quality >= 0.8);
});

ChatResponse response =
await client.GetResponseAsync("Summarize this contract.");
```

Provider failover can be composed within each quality tier:

```csharp
IChatClient cheapTier = new OrderedFailoverChatClient(
[cheapPrimaryClient, cheapBackupClient]);

IChatClient strongTier = new OrderedFailoverChatClient(
[strongPrimaryClient, strongBackupClient]);

using IChatClient client = new QualityCascadeChatClient(
clients: [cheapTier, strongTier],
isResponseAccepted: EvaluateResponseAsync);

ChatResponse response =
await client.GetResponseAsync("Summarize this contract.");
```

In this composition, each inner `OrderedFailoverChatClient` handles provider failures for its tier. The outer `CascadingChatClient` evaluates successful responses and handles quality escalation between tiers.

The streaming API remains available as a completed-response facade:

```csharp
await foreach (ChatResponseUpdate update in
client.GetStreamingResponseAsync("Summarize this contract."))
{
Console.Write(update.Text);
}
```

The first update is produced only after a complete response has been accepted.

### Alternative Designs

#### Extend `FailoverChatClient`

The original proposal added `FailoverChatClientAttempt.Response`, `RoutingContext.AttemptNumber`, and `FailoverChatClient.ShouldSelectAgainAsync`.

That design could reuse the existing failover loop, but it mixes two different completion models. Failover completion depends on provider failures, cancellation, attempt limits, and streaming output commitment. Cascading completion depends on a quality decision over a successful response.

The original source-code observations remain valid:

1. `FailoverChatClientAttempt` does not expose the completed `ChatResponse`.
2. A successful failover attempt is terminal before `OnRoutingUpdateAsync` runs.
3. `FailoverChatClient.GetResponseAsync` and `GetStreamingResponseAsync` are sealed overrides.

These facts explain why a subclass workaround is not available, but the revised proposal does not require changes to the failover APIs.

#### Add an attempt number to `RoutingContext`

Adding `AttemptNumber` would expose policy state on the general routing context even though one-shot routing does not need it. Internal request state, following the `OrderedFailoverChatClient` pattern, keeps the base abstraction focused.

The ordered client list is also the complete attempt budget. Each client is selected at most once, so a quality predicate that always rejects cannot create an unbounded loop.

#### Buffer native streams

A buffer-before-commit implementation could consume provider updates, reconstruct a `ChatResponse`, evaluate it, and then replay or discard the updates. The final tier could stream live because no further quality escalation is possible.

This works, but accepted early-tier responses still produce no output until completion. It adds buffering, response reconstruction, enumerator, and commitment complexity without preserving the main latency benefit of streaming. Using the non-streaming path and `ToChatResponseUpdates()` is a clearer initial contract.

#### Keep cascading in application code

Applications can implement this loop today because `RoutingChatClient` leaves its invocation methods virtual. That is useful for experimentation, but each implementation must handle concurrent request state, cancellation, client ownership, disposal, and streaming behavior correctly.

A shared `CascadingChatClient` removes that repeated infrastructure without prescribing how applications measure quality.

### Risks

#### Paid but discarded responses

A rejected response still consumes provider resources, tokens, and latency. Applications should measure escalation rate, discarded token usage, evaluator cost, and total request latency before assuming that a cascade is cheaper.

#### Streaming latency

The streaming facade does not provide live time to first token. Its first update is available only after a complete response has been accepted. This needs to be explicit so callers do not mistake update-shaped output for native streaming.

#### Capability differences between tiers

Clients in a ladder must be compatible with the requests routed to them. A tier without required tool calling, structured output, or message-history support cannot transparently replace another tier. The cascade does not normalize provider capabilities.

#### Quality-policy behavior

An evaluator may be slow, expensive, nondeterministic, or fail. Those effects remain part of the application policy. Evaluation failures are propagated instead of being silently converted into tier changes.

#### Final-response policy

The final response is returned even when an external judge might reject it. This guarantees finite progress through a fixed ladder. Applications requiring a strict minimum can validate the final response or use a domain-specific fallback as the last client.

#### Client lifetime

The cascade owns its clients by default. Applications that share client instances elsewhere must pass `leaveOpen: true` and manage those clients themselves.

Start by reading the existing RoutingChatClient and OrderedFailoverChatClient implementations, especially their request-state, disposal, and streaming behavior. Compare the proposal's CascadingChatClient contract with those entry points and the quality-cascade proof of concept. Done means the API behavior, concurrent request handling, client ownership, cancellation, and completed-response streaming facade are specified and covered by repository tests.

Written by the indexing model from the issue text.