[API Proposal]: Sliding-window retry budget and selectable defaults for the standard HTTP resilience handler

[Alejon]

Background and motivation

Background and motivation

AddStandardResilienceHandler() ships defaults that optimize the resilience of an individual
request/dependency relationship, but that can measurably reduce the resilience of the overall
system when a dependency is degraded:

• Retry.MaxRetryAttempts = 3 means a dependency that is failing can receive up to 4x its
  normal request volume at exactly the moment it is least able to serve it — classic retry
  amplification. Retry effectiveness also falls off sharply after the first attempt. Measured
  across a high-volume production service calling a large HTTP dependency, the success rate by
  retry number was roughly 26% / 3% / 2% — the 2nd and 3rd retries buy only a few percent of
  additional success for 3x the load.
• CircuitBreaker.FailureRatio = 0.1 opens the breaker while a dependency is still serving 90%
  of traffic successfully. In a recent multi-hour incident where a dependency's availability sat
  at 80–90%, a 10% breaker would have converted a partial degradation into a full outage on the
  caller side.

The service in question fans out to dozens of backend dependencies, so these defaults compound.
We shipped an in-house layer over the standard handler that (a) caps retries at one,
(b) floors the breaker at 50%, (c) gates retries of non-idempotent methods on ambiguous outcomes,
and (d) enforces a sliding-window retry budget capping retries at a fraction of request
volume per dependency. The budget is the piece with no equivalent anywhere in the stack, and the
one we think belongs in the framework.

A retry budget is well-established prior art outside .NET: Google SRE's client-side
retry_budget (10% recommended), Envoy's retry budgets, and gRPC's retryThrottling
(maxTokens / tokenRatio). It is strictly better than a fixed MaxRetryAttempts cap because
it bounds the aggregate amplification a caller can inflict, rather than the per-request one.

Today .NET has no way to express this:

• Polly v8 has no retry-budget concept (App-vNext/Polly#2348,
  closed not_planned). RetryStrategyOptions exposes only MaxRetryAttempts, BackoffType,
  UseJitter, Delay, MaxDelay, ShouldHandle, DelayGenerator, OnRetry, Randomizer.
• HttpStandardResilienceOptions has no plug point for a shared, stateful, per-named-client
  component consulted on each retry decision. The circuit breaker has ManualControl and
  StateProvider; retry has nothing equivalent.
• The only way to build one today is to capture and wrap ShouldHandle yourself, resolve a
  shared per-client budget instance out of DI, and add an outer DelegatingHandler to count the
  denominator (retries must not inflate their own denominator). That is a lot of machinery for
  something every high-fan-out service needs, and it is easy to get subtly wrong.
• RateLimiter is not a substitute: it caps concurrency for all traffic, not the ratio of
  retries to requests.

Separately, we would like a supported way to opt into a more conservative set of standard-handler
defaults without hand-tuning five strategies per client, and without taking a breaking change to
the existing defaults.

We have a reference implementation running in production behind rollout flags: a bucketed
sliding-window budget with a lock-free hot path (Interlocked + Volatile), lazy bucket reset,
and a monotonic TimeProvider clock. Benchmarked at sub-ns to low-ns per decision, 0 B
allocated; end-to-end retry pipeline overhead measured at ~5%, which is dwarfed by the load
it removes. We would be glad to contribute it.

API Proposal

API Proposal

1. Retry budget (primary ask)

namespace Microsoft.Extensions.Http.Resilience;

public class HttpRetryStrategyOptions : Polly.Retry.RetryStrategyOptions<HttpResponseMessage>
{
    // EXISTING: public bool ShouldRetryAfterHeader { get; set; }

    /// <summary>
    /// Gets or sets the retry budget that caps retry volume as a fraction of request volume.
    /// When <see langword="null" /> (the default) no budget is enforced.
    /// </summary>
    public HttpRetryBudgetOptions? RetryBudget { get; set; }
}

/// <summary>
/// Caps the volume of retries a client may issue to a fraction of the request volume observed
/// over a sliding window, preventing retry amplification against a degraded dependency.
/// </summary>
public class HttpRetryBudgetOptions
{
    /// <summary>
    /// Gets or sets the maximum ratio of retries to requests permitted within the sampling
    /// window. Default: 0.1.
    /// </summary>
    [Range(0.0, 1.0)]
    public double RetryRatio { get; set; } = 0.1;

    /// <summary>
    /// Gets or sets the duration over which request and retry volume is sampled.
    /// Default: 30 seconds.
    /// </summary>
    [TimeSpan("1s", "1d")]
    public TimeSpan SamplingDuration { get; set; } = TimeSpan.FromSeconds(30);

    /// <summary>
    /// Gets or sets the minimum number of requests that must be observed in the window before
    /// the ratio is enforced. Below this volume retries are always permitted. Default: 100.
    /// </summary>
    [Range(1, int.MaxValue)]
    public int MinimumThroughput { get; set; } = 100;
}

Naming intentionally mirrors CircuitBreakerStrategyOptions (FailureRatio, SamplingDuration,
MinimumThroughput) so the two rolling-window strategies read the same way. Bucket granularity is
an implementation detail and is deliberately not exposed.

Budget state is scoped to the resilience pipeline (i.e. one budget per named HttpClient), so a
failing dependency cannot consume another dependency's allowance. Requests are counted once per
logical request, not once per attempt.

2. Alternate defaults (secondary ask)

namespace Microsoft.Extensions.Http.Resilience;

/// <summary>
/// Identifies a named set of defaults for the standard resilience handler.
/// </summary>
public enum HttpResilienceDefaults
{
    /// <summary>
    /// The defaults that have always applied: 3 retries, a circuit breaker that opens at a 10%
    /// failure ratio, and retries for all HTTP methods.
    /// </summary>
    Standard = 0,

    /// <summary>
    /// Defaults tuned to minimize the load a caller adds to a degraded dependency: at most one
    /// retry, a retry budget, a circuit breaker that tolerates partial degradation, and retry
    /// suppression for non-idempotent methods whose outcome is unknown. Intended for services
    /// with high dependency fan-out.
    /// </summary>
    Conservative = 1,
}

public static class ResilienceHttpClientBuilderExtensions
{
    // EXISTING:
    // public static IHttpStandardResiliencePipelineBuilder AddStandardResilienceHandler(this IHttpClientBuilder builder);
    // public static IHttpStandardResiliencePipelineBuilder AddStandardResilienceHandler(this IHttpClientBuilder builder, Action<HttpStandardResilienceOptions> configure);
    // public static IHttpStandardResiliencePipelineBuilder AddStandardResilienceHandler(this IHttpClientBuilder builder, IConfigurationSection section);

    /// <summary>
    /// Adds the standard resilience handler using the specified named set of defaults.
    /// </summary>
    public static IHttpStandardResiliencePipelineBuilder AddStandardResilienceHandler(
        this IHttpClientBuilder builder,
        HttpResilienceDefaults defaults);
}

HttpResilienceDefaults.Conservative resolves to:

Setting	Standard	Conservative
Retry.MaxRetryAttempts	3	1
Retry.RetryBudget	null	enabled at defaults
CircuitBreaker.FailureRatio	0.1	0.5
Retry on ambiguous outcome, non-idempotent method	yes	no (see below)
RateLimiter, TotalRequestTimeout, AttemptTimeout	—	unchanged

The overload returns the existing IHttpStandardResiliencePipelineBuilder, so per-client overrides
and IConfigurationSection binding compose on top of the preset exactly as they do today — see
usage examples 2 and 3.

Naming and shape follow JsonSerializerDefaults (Web / General), which solves the same problem
in the BCL: a named, discoverable set of defaults selected at the call site, extensible with new
values rather than new methods.

Standard = 0 is included so the existing behavior has a name and so the two options read as a
deliberate choice at the call site, but it is redundant with the parameterless overload and we have
no preference about whether it ships. Dropping it and having the enum carry only Conservative is
equally fine by us.

The idempotency behavior is intentionally not exposed as standalone public API. Under
Conservative, retries are suppressed for non-idempotent methods (POST, PATCH) only when the
outcome is ambiguous — that is, when no response was received and it is unknown whether the server
processed the request. Definitive rejections such as 429 or 503 remain retriable for all methods,
since the server told us it did not process the request.

This is deliberately narrower than the existing DisableForUnsafeHttpMethods(), which is too
coarse for us in two ways: it also excludes PUT and DELETE (idempotent per RFC 9110 §9.2.2), and it
suppresses retries even for definitive 429/503 responses where no duplicate side effect is
possible. See dotnet/extensions#5248. We are ok with using it as part of Conservative as well though if having different behavior is a deal breaker.

Bundling it into the preset rather than shipping a second HttpRetryStrategyOptionsExtensions
method keeps the total new surface to one options class, one property, one enum, and one overload.
If the team would rather have it as an independently callable knob, we are happy to split it out.

API Usage

// 1. Opt a single client into a retry budget on top of the standard handler.
services.AddHttpClient("inventory")
    .AddStandardResilienceHandler(options =>
    {
        options.Retry.RetryBudget = new HttpRetryBudgetOptions
        {
            RetryRatio = 0.05,                             // at most +5% load from retries
            SamplingDuration = TimeSpan.FromSeconds(30),
            MinimumThroughput = 100,
        };
    });

// 2. Apply the preset, then override one knob.
services.AddHttpClient("pricing")
    .AddStandardResilienceHandler(HttpResilienceDefaults.Conservative)
    .Configure(options => options.Retry.RetryBudget!.RetryRatio = 0.02);

// 3. Apply the preset, then bind overrides from configuration.
services.AddHttpClient("catalog")
    .AddStandardResilienceHandler(HttpResilienceDefaults.Conservative)
    .Configure(configuration.GetSection("HttpResilience:Catalog"));

{
  "HttpResilience": {
    "Catalog": {
      "Retry": {
        "MaxRetryAttempts": 1,
        "RetryBudget": {
          "RetryRatio": 0.05,
          "SamplingDuration": "00:00:30",
          "MinimumThroughput": 100
        }
      },
      "CircuitBreaker": { "FailureRatio": 0.5 }
    }
  }
}

// 4. Retry-only pipeline (no circuit breaker) still gets the budget.
services.AddHttpClient("legacy")
    .AddResilienceHandler("legacy-retry", (pipeline, _) =>
        pipeline.AddRetry(new HttpRetryStrategyOptions
        {
            MaxRetryAttempts = 1,
            RetryBudget = new HttpRetryBudgetOptions { RetryRatio = 0.05 },
        }));

// 5. What the Conservative preset changes about retry eligibility.
services.AddHttpClient("orders")
    .AddStandardResilienceHandler(HttpResilienceDefaults.Conservative);

// POST /orders that times out          -> NOT retried (server may have created the order)
// POST /orders that returns 503        -> retried (server definitively rejected it)
// PUT  /orders/42 that times out       -> retried (idempotent)
// GET  /orders/42, budget exhausted    -> NOT retried (budget)

Alternative Designs

1. Change the shipped defaults instead of adding API. Lower MaxRetryAttempts to 1 and raise
   FailureRatio to 0.5 in HttpStandardResilienceOptions. This is what we actually believe is
   correct for most services, and it needs no new API — but it is a behavioral breaking change for
   everyone on AddStandardResilienceHandler, so we are not proposing it as the primary path. If
   the team is open to it, we would support it for the next major version, ideally with the budget
   available first so callers have a migration story. Happy to share the telemetry behind the
   recommendation.

2. Ship the budget in Polly instead of Microsoft.Extensions.Http.Resilience. Arguably the
   more natural home, since a retry budget is transport-agnostic. Polly declined this
   (#2348). If that position has changed, a
   RetryStrategyOptions<T>.RetryBudget in Polly would let this proposal shrink to just the
   HTTP-specific defaults preset.

3. Expose an extensibility hook rather than a concrete budget. For example
   Func<RetryPredicateArguments<HttpResponseMessage>, ValueTask<bool>> RetryFilter plus a
   documented pattern, or a RetryBudgetProvider abstraction. More flexible, but pushes the hard
   parts — correctly counting the denominator once per logical request, lock-free windowing,
   per-client instance scoping — back onto every consumer. Those are exactly the parts we got
   wrong on our first attempt.

4. Make RetryBudget an abstract/pluggable type (HttpRetryBudget base class with a built-in
   sliding-window implementation) so callers can supply e.g. a distributed or cross-client shared
   budget. More extensible; more surface area. We would be happy with either, but suspect a
   sealed options class covers the overwhelming majority of cases.

5. Document the ShouldHandle-wrapping recipe instead of adding API. Cheapest option for the
   team. We think a correct implementation is subtle enough (denominator counting, instance
   scoping, allocation-free hot path, TimeProvider for testability) that guidance alone will
   produce mostly-broken copies.

6. Where the preset is selected. Instead of an AddStandardResilienceHandler overload, the
   named defaults could be applied via an HttpStandardResilienceOptions constructor —
   new HttpStandardResilienceOptions(HttpResilienceDefaults.Conservative) — which is the exact
   JsonSerializerOptions parallel. We did not propose it because the options object is produced by
   the options pattern rather than constructed by the caller, so a constructor is not reachable on
   the normal path. An ApplyDefaults(HttpResilienceDefaults) instance method on the options would
   also work and composes inside an existing Configure callback; the builder overload was chosen
   because it puts the choice at the call site where the handler is added, which is more
   discoverable.

Risks

Not a breaking change as proposed. RetryBudget defaults to null and the preset is
opt-in, so existing behavior is unchanged. Alternative Design #1 is breaking and is called
out as such.

• Hot-path cost. The budget is consulted on every retry decision. Our implementation measures
  in the low nanoseconds with zero allocation (Interlocked/Volatile over a fixed bucket array,
  no locks on the fast path); end-to-end retry pipeline overhead was ~5%. A naive lock-based
  implementation would be much worse, which is part of the argument for shipping one rather than
  documenting one.
• Silently suppressed retries are hard to debug. A budget makes retry behavior depend on
  aggregate traffic, so the same request can behave differently under load. This needs
  first-class observability: a distinct outcome on the existing resilience telemetry (e.g. a
  retry-budget-exhausted reason) and a counter, not just a log line. We consider this a
  requirement of the feature, not a nice-to-have.
• Misconfiguration can disable retries entirely. RetryRatio = 0 with a low
  MinimumThroughput silently stops all retries. Mitigated by the [Range]/[TimeSpan]
  validation attributes above and by the MinimumThroughput low-traffic escape hatch, which
  exists because floor(RetryRatio * requests) rounds to 0 at low volume and would otherwise
  block the very first retry.
• Budget scoping is a semantic decision. Per named HttpClient is the right default, but a
  service that reaches one dependency through several named clients will under-count that
  dependency's aggregate load. Worth deciding whether a shared/named budget is in scope for v1.
• Interaction with MaxRetryAttempts and the circuit breaker must be specified: the budget
  can only further restrict retries the existing predicate already approved, and budget-denied
  retries should not be recorded as circuit-breaker failures. Our implementation follows both
  rules; the docs need to state them.
• Enum values are a compatibility surface. Once HttpResilienceDefaults.Conservative ships,
  changing what it resolves to is a silent behavior change for everyone who selected it — the same
  constraint JsonSerializerDefaults lives under. The values should be treated as frozen on
  release, with new presets added as new enum values rather than by re-tuning existing ones.

[rafal-mz]

I very much like this proposal, and how its rooted in real production data at scale.

From what I see, the Polly authors rejected the ask to add error budget since they were worried about the mechanics of the change and how time consuming it would be. I think, if you are willing to do it at the Polly layer, it would compose much better. It seems like a good fit to be just an extension point for retry handler.

As far as I remember, @martintmk is a Polly maintainer, so we can reach out for more architecture advises to him.

API suggestions

1. I would personally expose AddStandardResillienceHandlerV2() instead of having an enum. IMHO it would clarify for the callers that we are improving on the previous settings. If you still prefer to have an enum, I would swap conservative to still be the first option, and name the new one similarly to high-scale, alternative, modern, recommended.
2. If we are changing DisableForUnsafeHttpMethods under the hood, I would expose it the same way as the handler changes, e.g. DisableForUnsafeHttpMethodsV2.
3. I would personally start simple, and contribute an existing implementation that you've crafted based on your experience, and expose an extension point for error budget alternatives only when we get such a request.

Questions

Also, do I understand correct that with current standard resillience pipeline the rate limiting count does not count the retries on the downstream? I.e. each retry is executed independently of the rate limiter? If that's the case, if we limit the requests to 1000, and have 3 retries, it means the maximum fan out is ~2000 requests in couple of seconds? If so, did you find this problematic in reasoning about how much requests will you send to your downstream services?