Skip to content

[BUG] azure-core-amqp: token-manager close or terminal error can interrupt refresh and close the shared dispatcher pipe #51112

Open

Description

@j7nw4r

Problem

Closing ActiveClientTokenManager, or receiving a terminal authorization error while refresh initiation is still active, can interrupt the initiating thread. If that thread is in or subsequently enters ReactorDispatcher.signalWorkQueue, the interrupt can close the connection's shared NIO pipe with ClosedByInterruptException.

This is a separate lifecycle/transport-ownership issue found while investigating #50922 and reviewing #50923. It reproduces on both the original implementation and the proposed fix. Keep this follow-up separate from #50923, which addresses interruption caused by replacing the refresh interval.

The token manager belongs to a link's authorization lifetime; the dispatcher pipe belongs to the shared connection. Closing one link or terminally failing its authorization must not damage a healthy transport still used by sibling links. Both Java Event Hubs and Service Bus use this core implementation. Sibling-link disruption is source-traced, not yet reproduced against a live service.

Relevant implementation:

Reproduction and evidence

An independent local probe uses the actual ActiveClientTokenManager, ReactorDispatcher, Proton reactor, and NIO pipe, with a controlled CBS node. The probe never calls Thread.interrupt() and never injects ClosedByInterruptException.

  1. Complete initial authorization with a short token lifetime to schedule a refresh.
  2. Hold the refresh-initiating thread inside the CBS subscription callback using explicit coordination.
  3. From another thread, either call tokenManager.close() or complete that CBS request with a non-transient error.
  4. Release the initiating thread and invoke the real dispatcher from that thread.
  5. Observe its interrupt state, pipe state, and dispatcher shutdown cause.
Trigger during active refresh initiation Main snapshot ebf8519e8f0 PR #50923 head d56e5c38df0
Explicit token-manager close Thread interrupted; pipe closed Thread interrupted; pipe closed
Terminal authorization error Thread interrupted; pipe closed Thread interrupted; pipe closed
Successful cross-thread reply / interval rescheduling (control) Thread interrupted; pipe closed No interruption; pipe remains open

Representative output for both lifecycle cases:

interrupted=true pipeOpen=false shutdownCause=java.nio.channels.ClosedByInterruptException calls=2

Environment: Microsoft OpenJDK 21.0.7, macOS aarch64, Reactor 3.8.7. Tests compiled the exact target token-manager source over released core-AMQP supporting dependencies. These are focused probe results, not full-module build results. Main's token-manager and dispatcher sources were also confirmed byte-identical to the 2.12.1 source release, but the lifecycle matrix above was run with Reactor 3.8.7.

The overlap is deliberately forced. The dispatcher call occurs after the controlled cancellation/error, so this proves cancellation-induced interruption and actual pipe closure; it does not constitute a full CBS/network exchange. Production incidence, live connection recovery, Java 17 behavior, and sibling-link effects still require validation.

Standalone Java probe used in the investigation

Compile this with the selected core-AMQP implementation and its runtime dependencies on the classpath. Run RefreshInterruptProbe true close and RefreshInterruptProbe true terminal. The true argument asserts that the current bug is observed; after fixing the lifecycle issue, use false to assert no interruption or dispatcher error.

import com.azure.core.amqp.ClaimsBasedSecurityNode;
import com.azure.core.amqp.implementation.ActiveClientTokenManager;
import com.azure.core.amqp.implementation.ReactorDispatcher;
import java.nio.channels.Pipe;
import java.time.OffsetDateTime;
import java.time.Duration;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import org.apache.qpid.proton.Proton;
import org.apache.qpid.proton.reactor.Reactor;
import reactor.core.publisher.Mono;
import reactor.core.publisher.MonoSink;

public class RefreshInterruptProbe {
  public static void main(String[] args) throws Exception {
    boolean expectInterrupted = args.length == 0 || Boolean.parseBoolean(args[0]);
    String mode = args.length < 2 ? "success" : args[1];
    Pipe pipe = Pipe.open();
    Reactor reactor = Proton.reactor();
    ReactorDispatcher dispatcher = new ReactorDispatcher("independent-refresh-probe", reactor, pipe);
    AtomicReference<Throwable> shutdown = new AtomicReference<>();
    dispatcher.getShutdownSignal().subscribe(x -> {}, shutdown::set);
    AtomicInteger calls = new AtomicInteger();
    AtomicReference<MonoSink<OffsetDateTime>> refreshSink = new AtomicReference<>();
    AtomicBoolean release = new AtomicBoolean();
    AtomicBoolean interrupted = new AtomicBoolean();
    CountDownLatch entered = new CountDownLatch(1), finished = new CountDownLatch(1);
    ClaimsBasedSecurityNode node = new ClaimsBasedSecurityNode() {
      public Mono<OffsetDateTime> authorize(String a, String s) {
        if (calls.incrementAndGet() == 1) return Mono.just(OffsetDateTime.now().plusSeconds(3));
        return Mono.create(sink -> {
          refreshSink.set(sink);
          entered.countDown();
          if (mode.equals("same")) { sink.success(OffsetDateTime.now().plusHours(1)); release.set(true); }
          // Model the initiating thread still inside CBS dispatch while a reply completes elsewhere.
          long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(5);
          while (!release.get() && System.nanoTime() < deadline) Thread.onSpinWait();
          interrupted.set(Thread.currentThread().isInterrupted());
          try { dispatcher.invoke(() -> {}); } catch (Exception e) { throw new RuntimeException(e); }
          finally { finished.countDown(); }
        });
      }
      public void close() {}
    };
    ActiveClientTokenManager manager = new ActiveClientTokenManager(Mono.just(node), "audience", "scope");
    manager.authorize().block(Duration.ofSeconds(2));
    if (!entered.await(3, TimeUnit.SECONDS)) throw new AssertionError("No refresh");
    if (mode.equals("same")) { /* source completes on its own thread */ }
    else if (mode.equals("close")) manager.close();
    else if (mode.equals("terminal")) refreshSink.get().error(new IllegalStateException("injected terminal CBS failure"));
    else if (mode.equals("transient")) refreshSink.get().error(new com.azure.core.amqp.exception.AmqpException(true, "injected transient CBS failure", new com.azure.core.amqp.exception.AmqpErrorContext("test")));
    else refreshSink.get().success(OffsetDateTime.now().plusHours(1));
    if (!mode.equals("same")) release.set(true);
    if (!finished.await(3, TimeUnit.SECONDS)) throw new AssertionError("Refresh did not return");
    Throwable cause = shutdown.get() == null ? null : shutdown.get().getCause();
    System.out.printf("interrupted=%s pipeOpen=%s shutdownCause=%s calls=%d%n",
      interrupted.get(), pipe.sink().isOpen(), cause, calls.get());
    manager.close(); pipe.sink().close(); pipe.source().close(); reactor.free();
    if (interrupted.get() != expectInterrupted) throw new AssertionError("Unexpected interruption");
    if (expectInterrupted && !(cause instanceof java.nio.channels.ClosedByInterruptException))
      throw new AssertionError("Did not reproduce actual dispatcher pipe failure");
    if (!expectInterrupted && cause != null) throw new AssertionError("Dispatcher failed");
  }
}

Required behavior and regression coverage

  • Define cancellation ownership so ending one link's authorization lifetime cannot corrupt shared dispatcher I/O. Cancellation may stop link-specific work; it must preserve the healthy shared transport.
  • Add deterministic tests for close and terminal error while refresh dispatch is active. Verify the shared pipe stays usable and the original terminal error is handled appropriately.
  • Add a test with sibling links sharing a connection: close or terminally fail one link during renewal while the others continue sending/receiving. Assert continued sibling progress without an unexpected shared-connection shutdown.
  • Verify that no new authorization starts for a closed token manager and that workers/subscriptions are released across repeated cycles.
  • Exercise the external renewal/close/forward-progress invariant for both Event Hubs and Service Bus. Keep the Java-specific forced overlap test in core-AMQP coverage.

This follow-up does not require broadening the interval-rescheduling fix in #50923.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Azure.Core.AMQPazure-core-amqpClientThis issue points to a problem in the data-plane of the library.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions