Skip to content

[BUG] RequestRetryOptions tryTimeout has no effect on synchronous Storage clients #51144

Open

Description

@willwwt

Describe the bug
For synchronous Storage clients (for example BlobServiceClient), the per-try timeout configured with RequestRetryOptions (tryTimeout) never fires. A stalled request runs until the HTTP client's own timeout (60 s by default for Netty) instead of the configured try timeout. The async client with the same options times out as expected.

Root cause
RequestRetryPolicy.attemptSync runs each try as:

Mono<HttpResponse> httpResponseMono = Mono.fromCallable(() -> next.clone().processSync());
httpResponseMono = httpResponseMono.timeout(this.requestRetryOptions.getTryTimeoutDuration());
HttpResponse response = httpResponseMono.block();

Mono.fromCallable runs the blocking processSync() on the subscribing thread inside request(n). In FluxTimeout.TimeoutMainSubscriber.onSubscribe, actual.onSubscribe(this) (which triggers request(n) and therefore the whole blocking call) runs before firstTimeout.subscribe(timeoutSubscriber) arms the timer. By the time the timer is armed, the value has already been emitted, so the timeout can never fire. attemptAsync is not affected because process() does not block on subscribe.

Stack of a sync call blocked on a stalled server (trimmed):

reactor.core.publisher.Mono.block(Mono.java:1779)
com.azure.core.http.netty.NettyAsyncHttpClient.sendSync(NettyAsyncHttpClient.java:194)
...
com.azure.storage.common.policy.RequestRetryPolicy.lambda$attemptSync$0(RequestRetryPolicy.java:266)
reactor.core.publisher.MonoCallable$MonoCallableSubscription.request(MonoCallable.java:137)
reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2330)
reactor.core.publisher.SerializedSubscriber.request(SerializedSubscriber.java:151)

To Reproduce
Point a client at a server that accepts the TCP connection but never responds (a ServerSocket that is never accept()ed is enough), with maxTries = 1 and tryTimeout = 1s.

Code Snippet

import com.azure.storage.blob.BlobServiceClientBuilder;
import com.azure.storage.common.StorageSharedKeyCredential;
import com.azure.storage.common.policy.RequestRetryOptions;
import com.azure.storage.common.policy.RetryPolicyType;
import java.net.ServerSocket;
import java.time.OffsetDateTime;

public class TryTimeoutRepro {
  public static void main(String[] args) throws Exception {
    // The listen backlog completes the TCP handshake, but nothing is ever written back.
    try (ServerSocket stalled = new ServerSocket(0)) {
      BlobServiceClientBuilder builder = new BlobServiceClientBuilder()
          .endpoint("http://127.0.0.1:" + stalled.getLocalPort() + "/devstoreaccount1")
          .credential(new StorageSharedKeyCredential("devstoreaccount1",
              "Eby8vdM02xNOcqCfGfP/mkP3+rZTt5tF5If1fMvj08QzZLOiT4E5lZaz0smU4wFwbIsrpPoIQx=="))
          .retryOptions(new RequestRetryOptions(RetryPolicyType.EXPONENTIAL,
              1 /* maxTries */, 1 /* tryTimeoutInSeconds */, null, null, null));
      OffsetDateTime start = OffsetDateTime.now();

      long t0 = System.nanoTime();
      try {
        builder.buildAsyncClient().getUserDelegationKey(start, start.plusHours(1)).block();
      } catch (RuntimeException e) {
        System.out.println("async: " + e.getCause() + " after " + ms(t0) + " ms");
      }

      t0 = System.nanoTime();
      try {
        builder.buildClient().getUserDelegationKey(start, start.plusHours(1));
      } catch (RuntimeException e) {
        System.out.println("sync: " + e + " after " + ms(t0) + " ms");
      }
    }
  }

  private static long ms(long t0) {
    return (System.nanoTime() - t0) / 1_000_000;
  }
}

Observed with an equivalent test:

  • async: TimeoutException after ~1009 ms
  • sync: still running after 5000 ms; each try only ends at the HTTP client's 60 s response timeout (two tries took ~120 s)

The same behavior in plain Reactor:

Mono.fromCallable(() -> { Thread.sleep(4000); return "OK"; })
    .timeout(Duration.ofSeconds(1))
    .block();                                   // returns "OK" after ~4001 ms, no timeout

Mono.fromCallable(() -> { Thread.sleep(4000); return "OK"; })
    .subscribeOn(Schedulers.boundedElastic())
    .timeout(Duration.ofSeconds(1))
    .block();                                   // TimeoutException after ~1003 ms

Expected behavior
Each sync try fails with a timeout after tryTimeout and is retried, the same as on the async client.

Possible fix
Run the blocking call off the subscribing thread (for example subscribeOn(Schedulers.boundedElastic())), or enforce the timeout with an executor and Future.get(timeout). If changing this is not desirable, documenting that tryTimeout does not apply to sync clients would help.

Workaround
Set connect, response and read timeouts on the HTTP client (NettyAsyncHttpClientBuilder) and pass it with .httpClient(...), or use the async client.

Setup

  • OS: Ubuntu 20.04 (x86_64)
  • Library/Libraries: azure-storage-blob 12.36.0, azure-storage-common 12.35.0, azure-core 1.60.0, azure-core-http-netty 1.16.8, reactor-core 3.7.19, reactor-netty-http 1.2.18. The code in RequestRetryPolicy.attemptSync is unchanged on main.
  • Java version: 25 (GraalVM 25.0.4)
  • Frameworks: none

Information Checklist

  • Bug Description Added
  • Repro Steps Added
  • Setup information Added

Activity

  1. github-actions commented on Nov 6, 2026

    @github-actions
    Contributor

    Thank you for your feedback. Tagging and routing to the team member(s) best able to assist.

    //cc: xgithubtriage

  2. github-actions commented on Nov 6, 2026

    @github-actions
    Contributor

    🎯 Agentic Issue Triage

    Summary: Sync Storage clients never honor RequestRetryOptions tryTimeout, because RequestRetryPolicy.attemptSync wraps a blocking processSync() in Mono.fromCallable(...).timeout(...), so the timer is armed only after the blocking call returns.

    📋 Issue Details

    Package: com.azure:azure-storage-common / azure-storage-blob (version not stated)
    Affected API: RequestRetryPolicy.attemptSync, e.g. BlobServiceClient.getUserDelegationKey
    Scenarios:

    • Server accepts TCP but never responds; maxTries=1, tryTimeout=1s
    • Async client times out at ~1s; sync client runs until the HTTP client timeout (60s for Netty)

    Root ask: Make the per-try timeout effective for sync clients.

    🔎 Debugging / Reproduction Notes

    The reporter supplied a clear repro and root-cause analysis (timeout subscribed after request(n) triggers the blocking call).

    Suggested investigation steps:

    1. Reproduce with the provided snippet.
    2. Run the sync try on a separate thread/executor (or use an HTTP-client-level timeout) so the timeout can interrupt it.
    3. Add a sync stalled-server test and check other sync policies using the same pattern.
    🏷️ Label Confidence
    • Category: Client — SDK code behavior.
    • Service: Storage — storage common retry policy / blob client.
    • Confidence: High — package and component are explicitly named.
    👥 Owner Routing
    • Matched CODEOWNERS entry: # ServiceLabel: %Storage (line 466)
    • AzureSdkOwners: none listed
    • ServiceOwners: @xgithubtriage
    • Routing action: Requested Service Attention and needs-team-attention; left unassigned.
    • Scan notes: %Mgmt and other later entries did not match the predicted labels.

    Generated by Agentic Triage for #51144 · auto · 21.3 AIC · ⊞ 16.5K · ◷

  3. added
    questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that
    StorageStorage Service (Queues, Blobs, Files)
    Service AttentionWorkflow: This issue is responsible by Azure service team.
    ClientThis issue points to a problem in the data-plane of the library.
    customer-reportedIssues that are reported by GitHub users external to the Azure organization.
    needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK team
    on Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

ClientThis issue points to a problem in the data-plane of the library.Service AttentionWorkflow: This issue is responsible by Azure service team.StorageStorage Service (Queues, Blobs, Files)customer-reportedIssues that are reported by GitHub users external to the Azure organization.needs-team-attentionWorkflow: This issue needs attention from Azure service team or SDK teamquestionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions