[API Proposal]: Add CascadingChatClient for response-quality routing

[WittBen]

Background and motivation

Update (2026-08-22): This proposal was revised based on the discussion below. It now proposes a separate CascadingChatClient instead of extending FailoverChatClient.

The experimental routing APIs ([Experimental("MEAI001")]) introduced RoutingChatClient, SemanticRoutingChatClient, FailoverChatClient, and OrderedFailoverChatClient.

These types support selecting a client for a request and selecting another client after an eligible provider failure. What is still missing is an ordered routing mode that can continue after a request succeeds but the application does not accept the response quality.

That is the model-cascading pattern described by approaches such as FrugalGPT: start with a less expensive model, evaluate the completed response, and use a stronger model only when necessary.

This proposal does not define how quality is measured. That decision can remain in the application or evaluation layer and may use a predicate, a confidence score, Microsoft.Extensions.AI.Evaluation, or another application-specific policy.

Cascading and failover have different completion semantics. Failover is driven by provider failures and, for streaming, whether output has already been committed. Cascading is driven by a quality decision over a successful, completed ChatResponse. Keeping them as separate routing clients gives each behavior a smaller and clearer contract.

The two behaviors can still be combined through composition. Each cascade rung is an IChatClient, so a rung can itself be a resilient FailoverChatClient. The inner client handles provider failures within a tier, while the outer CascadingChatClient handles quality escalation between tiers.

A working proof of concept is available here:

https://github.com/WittBen/meai-quality-cascade

API Proposal

namespace Microsoft.Extensions.AI;

[Experimental("MEAI001")]
public abstract class CascadingChatClient : RoutingChatClient
{
    protected CascadingChatClient(
        IReadOnlyList<IChatClient> clients,
        bool leaveOpen = false);

    protected abstract ValueTask<bool> IsResponseAcceptedAsync(
        RoutingContext context,
        IChatClient client,
        ChatResponse response,
        CancellationToken cancellationToken);
}

No changes are proposed for RoutingContext, FailoverChatClient, or FailoverChatClientAttempt. In particular, this revision no longer proposes RoutingContext.AttemptNumber, FailoverChatClientAttempt.Response, or FailoverChatClient.ShouldSelectAgainAsync.

Behavioral contract

The constructor receives a non-empty ordered list of chat clients. The list is snapshotted and must not contain null entries.

Each request creates its own RoutingContext and starts with the first configured client. After a client returns a successful ChatResponse:

• If another client remains, IsResponseAcceptedAsync is called.
• Returning true returns the response immediately.
• Returning false discards the response and invokes the next client.
• The final client's response is returned without another quality decision.

The final-response rule ensures that the finite client list always terminates. An application that requires a hard quality floor can validate the returned response outside the cascade.

The quality hook receives the request's RoutingContext, the client that produced the response, the completed ChatResponse, and the request cancellation token. This allows an application to apply request- or model-specific policy without introducing a judging abstraction into MEAI.

A provider exception, evaluation exception, or cancellation is propagated to the caller and does not cause quality escalation. Provider failover can be added by using a FailoverChatClient as an individual cascade rung.

The current rung is request-scoped and safe for concurrent requests. The implementation can follow the existing OrderedFailoverChatClient pattern and use a ConcurrentDictionary<RoutingContext, int>. Retained state is removed after completion, provider failure, evaluation failure, cancellation, or disposal.

CascadingChatClient does not provide native streaming. GetStreamingResponseAsync uses the non-streaming cascade path and converts the accepted response with ChatResponse.ToChatResponseUpdates(). The returned updates therefore do not provide native time-to-first-token latency.

The cascade owns and disposes its configured clients by default. Passing leaveOpen: true leaves their lifetime with the caller.

API Usage

A small concrete implementation can keep the quality policy in application code:

internal sealed class QualityCascadeChatClient : CascadingChatClient
{
    private readonly Func<
        RoutingContext,
        IChatClient,
        ChatResponse,
        CancellationToken,
        ValueTask<bool>> _isResponseAccepted;

    public QualityCascadeChatClient(
        IReadOnlyList<IChatClient> clients,
        Func<
            RoutingContext,
            IChatClient,
            ChatResponse,
            CancellationToken,
            ValueTask<bool>> isResponseAccepted,
        bool leaveOpen = false)
        : base(clients, leaveOpen)
    {
        _isResponseAccepted = isResponseAccepted;
    }

    protected override ValueTask<bool> IsResponseAcceptedAsync(
        RoutingContext context,
        IChatClient client,
        ChatResponse response,
        CancellationToken cancellationToken)
        => _isResponseAccepted(context, client, response, cancellationToken);
}

The application supplies the ordered clients and decides what an acceptable response means:

using IChatClient client = new QualityCascadeChatClient(
    clients: [cheapClient, strongClient],
    isResponseAccepted: static (context, selectedClient, response, cancellationToken) =>
    {
        cancellationToken.ThrowIfCancellationRequested();

        double quality =
            response.AdditionalProperties?.TryGetValue("quality", out object? value) == true
                ? Convert.ToDouble(value)
                : 0;

        return new ValueTask<bool>(quality >= 0.8);
    });

ChatResponse response =
    await client.GetResponseAsync("Summarize this contract.");

Provider failover can be composed within each quality tier:

IChatClient cheapTier = new OrderedFailoverChatClient(
    [cheapPrimaryClient, cheapBackupClient]);

IChatClient strongTier = new OrderedFailoverChatClient(
    [strongPrimaryClient, strongBackupClient]);

using IChatClient client = new QualityCascadeChatClient(
    clients: [cheapTier, strongTier],
    isResponseAccepted: EvaluateResponseAsync);

ChatResponse response =
    await client.GetResponseAsync("Summarize this contract.");

In this composition, each inner OrderedFailoverChatClient handles provider failures for its tier. The outer CascadingChatClient evaluates successful responses and handles quality escalation between tiers.

The streaming API remains available as a completed-response facade:

await foreach (ChatResponseUpdate update in
    client.GetStreamingResponseAsync("Summarize this contract."))
{
    Console.Write(update.Text);
}

The first update is produced only after a complete response has been accepted.

Alternative Designs

Extend FailoverChatClient

The original proposal added FailoverChatClientAttempt.Response, RoutingContext.AttemptNumber, and FailoverChatClient.ShouldSelectAgainAsync.

That design could reuse the existing failover loop, but it mixes two different completion models. Failover completion depends on provider failures, cancellation, attempt limits, and streaming output commitment. Cascading completion depends on a quality decision over a successful response.

The original source-code observations remain valid:

1. FailoverChatClientAttempt does not expose the completed ChatResponse.
2. A successful failover attempt is terminal before OnRoutingUpdateAsync runs.
3. FailoverChatClient.GetResponseAsync and GetStreamingResponseAsync are sealed overrides.

These facts explain why a subclass workaround is not available, but the revised proposal does not require changes to the failover APIs.

Add an attempt number to RoutingContext

Adding AttemptNumber would expose policy state on the general routing context even though one-shot routing does not need it. Internal request state, following the OrderedFailoverChatClient pattern, keeps the base abstraction focused.

The ordered client list is also the complete attempt budget. Each client is selected at most once, so a quality predicate that always rejects cannot create an unbounded loop.

Buffer native streams

A buffer-before-commit implementation could consume provider updates, reconstruct a ChatResponse, evaluate it, and then replay or discard the updates. The final tier could stream live because no further quality escalation is possible.

This works, but accepted early-tier responses still produce no output until completion. It adds buffering, response reconstruction, enumerator, and commitment complexity without preserving the main latency benefit of streaming. Using the non-streaming path and ToChatResponseUpdates() is a clearer initial contract.

Keep cascading in application code

Applications can implement this loop today because RoutingChatClient leaves its invocation methods virtual. That is useful for experimentation, but each implementation must handle concurrent request state, cancellation, client ownership, disposal, and streaming behavior correctly.

A shared CascadingChatClient removes that repeated infrastructure without prescribing how applications measure quality.

Risks

Paid but discarded responses

A rejected response still consumes provider resources, tokens, and latency. Applications should measure escalation rate, discarded token usage, evaluator cost, and total request latency before assuming that a cascade is cheaper.

Streaming latency

The streaming facade does not provide live time to first token. Its first update is available only after a complete response has been accepted. This needs to be explicit so callers do not mistake update-shaped output for native streaming.

Capability differences between tiers

Clients in a ladder must be compatible with the requests routed to them. A tier without required tool calling, structured output, or message-history support cannot transparently replace another tier. The cascade does not normalize provider capabilities.

Quality-policy behavior

An evaluator may be slow, expensive, nondeterministic, or fail. Those effects remain part of the application policy. Evaluation failures are propagated instead of being silently converted into tier changes.

Final-response policy

The final response is returned even when an external judge might reject it. This guarantees finite progress through a fixed ladder. Applications requiring a strict minimum can validate the final response or use a domain-specific fallback as the last client.

Client lifetime

The cascade owns its clients by default. Applications that share client instances elsewhere must pass leaveOpen: true and manage those clients themselves.

[jozkee]

@joshuajyue

[joshuajyue]

I think it'd be useful to define a CascadingChatClient to not support native streaming. Let's think through the buffer commit strategy:

1. Attempt N is buffered, the full response is retrieved
2. Passes/fails a quality gate. Let's say it passes.
3. At that point, the full response is already available. Replaying the buffered updates provides the appearance of streaming, but none of its latency benefits. It would be simpler and faster to return the complete response.

I think a good middle ground there is to use the non-streaming pathway to get a response for streaming, and return response.ToChatResponseUpdates().

[joshuajyue]

public abstract class CascadingChatClient : RoutingChatClient
{
    protected CascadingChatClient(
        IReadOnlyList<IChatClient> clients,
        bool leaveOpen = false);

    protected abstract ValueTask<bool> IsResponseAcceptedAsync(
        RoutingContext context,
        IChatClient client,
        ChatResponse response,
        CancellationToken cancellationToken);
}

No need for the new context object. We can use a ConcurrentDictionary to keep track of attempt number. Adding attempt number to RoutingContext would add attempt numbering to the base RoutingChatClient, which doesn't need it.
I think separating Failover and Cascading is also useful, given the streaming consideration earlier and also that currently, Failover's completion condition is whether an output was committed.

[WittBen]

Thanks, this is clarifying, and I think the separate CascadingChatClient is the cleaner shape. Reactions:

Separating Cascading from Failover. Agreed on the semantics. Failover’s terminal condition is “an output was committed”; a cascade’s is “committed and accepted.” Folding both into one type (which my ShouldSelectAgainAsync sketch did) blurs that. IsResponseAcceptedAsync(context, client, response, ct) is also a smaller and more honest surface than my three additive changes (Response on the attempt, AttemptNumber on the context, plus the hook), and taking a completed ChatResponse sidesteps the streaming Response == null ambiguity entirely. I’m on board.

The one thing I’d like to nail down is error-failover composition. My original motivation was a single client that both retries on provider error and escalates on low quality. CascadingChatClient : RoutingChatClient doesn’t inherit FailoverChatClient‘s retry-on-error. I think the clean answer under your design is composition: make each rung a resilient client (e.g. a FailoverChatClient over that tier’s providers) and let CascadingChatClient own only quality escalation across rungs, so error-resilience is per-tier/inner and quality is outer. Is that the intended model? If so I’m happy with it, it’s arguably a better separation of concerns, but I’d like it stated explicitly so the “both behaviors at once” case has a blessed pattern.

Attempt tracking. Agreed that AttemptNumber shouldn’t go on RoutingContext, it doesn’t belong on the base abstraction. A ConcurrentDictionary keyed on the per-request context works; the only thing to get right is entry lifecycle (remove on completion/exception so it doesn’t leak), which stays internal to CascadingChatClient.

Streaming. Agreed: non-streaming, with GetStreamingResponseAsync delegating to the non-streaming path and returning response.ToChatResponseUpdates(). One note for the record, not a request: the final rung is never rejectable, so it could be streamed live without buffering, the only place real streaming latency survives. Fine to leave out of the initial contract.

If this direction looks right, I’ll rework the proposal and the PoC around a CascadingChatClient with IsResponseAcceptedAsync instead of the FailoverChatClient extension points.

[WittBen]

Update: I’ve now reworked the issue description and the proof of concept around the separate CascadingChatClient design discussed above.

The implementation uses IsResponseAcceptedAsync, request-scoped state, the non-streaming ToChatResponseUpdates() facade, and the existing leaveOpen ownership pattern. I also documented per-tier FailoverChatClient composition as the proposed way to combine provider resilience with quality escalation.

Updated PoC: https://github.com/WittBen/meai-quality-cascade