Skip to content

[google-auth-library-java] Impersonated, user and STS token requests don't retry a dropped connection #14772

Open

Description

@tbc-tim

Summary

ServiceAccountCredentials retries an I/O failure on its token request. The impersonation, user-credential and STS token requests don't, and two of them wrap the failure in an exception whose class hides the cause. A stale keep-alive connection (SocketException: Unexpected end of file from server, EOFException, SSLHandshakeException: connection closed) therefore fails the caller's request outright. Callers can't recover it either: google-cloud-core decides retryability from the thrown exception's class, never its cause.

Environment details

  1. google-auth-library-java (oauth2_http)
  2. OS: macOS 27 locally, Ubuntu (GitHub Actions) in CI
  3. Java: 21 and 25
  4. google-auth-library-oauth2-http 1.54.0, google-cloud-bigquery 2.73.0, google-cloud-core 2.77.0

What each token request does with an I/O failure (1.54.0)

Credential Token request Retry handler What the caller receives
ServiceAccountCredentials refreshAccessToken HttpBackOffIOExceptionHandler (line 571) retried
ImpersonatedCredentials IAM generateAccessToken (line 639) none new IOException("Error requesting access token", e) (line 643)
UserCredentials refresh-token grant (line 297) none GoogleAuthException.createWithTokenEndpointIOException(e) (line 303)
ExternalAccountCredentials STS exchange via StsRequestHandler (line 122) none the raw IOException

ExternalAccountCredentials with a service_account_impersonation_url refreshes through an internal ImpersonatedCredentials, so it inherits the impersonation row.

GoogleAuthException marks itself retryable, but nothing downstream reads that flag. google-cloud-core's BaseServiceException.isRetryable(boolean, IOException) and google-cloud-bigquery's default ExceptionHandler both match on the exception's own class (SocketException, ConnectException and so on). Neither the wrapped IOException nor GoogleAuthException matches.

Steps to reproduce

  1. Build an ImpersonatedCredentials or UserCredentials on an HttpTransportFactory whose first request throws java.net.SocketException.
  2. Call refresh(). It fails on the first attempt.
  3. Do the same with ServiceAccountCredentials. It retries and succeeds.

Code example

AtomicInteger calls = new AtomicInteger();
MockHttpTransport dropsFirstRequest = new MockHttpTransport() {
  @Override
  public LowLevelHttpRequest buildRequest(String method, String url) {
    return new MockLowLevelHttpRequest(url) {
      @Override
      public LowLevelHttpResponse execute() throws IOException {
        if (calls.incrementAndGet() == 1) {
          throw new SocketException("Unexpected end of file from server");
        }
        return new MockLowLevelHttpResponse()
            .setContentType("application/json")
            .setContent("{\"access_token\":\"t\",\"expires_in\":3600,\"token_type\":\"Bearer\"}");
      }
    };
  }
};

UserCredentials user = UserCredentials.newBuilder()
    .setClientId("c").setClientSecret("s").setRefreshToken("r")
    .setHttpTransportFactory(() -> dropsFirstRequest)
    .build();

user.refresh(); // throws GoogleAuthException; calls.get() == 1, no retry

Stack trace

From a BigQuery query authenticated with ImpersonatedCredentials in CI:

java.io.IOException: Error requesting access token
  at com.google.auth.oauth2.ImpersonatedCredentials.refreshAccessToken(ImpersonatedCredentials.java)
  ...
Caused by: java.net.SocketException: Unexpected end of file from server
  at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java)
  ...

Suggested fix

Give the impersonation, user-credential and STS token requests the same HttpBackOffIOExceptionHandler that ServiceAccountCredentials already sets. All three requests can be repeated safely: minting a token has no side effects.

Additional information

We work around this in a BigQuery JDBC driver by wrapping each credential and retrying refreshAccessToken() when the cause chain shows a dropped connection: Two-Bear-Capital/tbc-bq-jdbc#356. With that in place, every credential type above recovers from a single dropped token request. A fix in the library would make the wrapper unnecessary.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions