Description
Summary
ServiceAccountCredentials retries an I/O failure on its token request. The impersonation, user-credential and STS token requests don't, and two of them wrap the failure in an exception whose class hides the cause. A stale keep-alive connection (SocketException: Unexpected end of file from server, EOFException, SSLHandshakeException: connection closed) therefore fails the caller's request outright. Callers can't recover it either: google-cloud-core decides retryability from the thrown exception's class, never its cause.
Environment details
- google-auth-library-java (oauth2_http)
- OS: macOS 27 locally, Ubuntu (GitHub Actions) in CI
- Java: 21 and 25
- google-auth-library-oauth2-http 1.54.0, google-cloud-bigquery 2.73.0, google-cloud-core 2.77.0
What each token request does with an I/O failure (1.54.0)
| Credential | Token request | Retry handler | What the caller receives |
|---|---|---|---|
ServiceAccountCredentials |
refreshAccessToken |
HttpBackOffIOExceptionHandler (line 571) |
retried |
ImpersonatedCredentials |
IAM generateAccessToken (line 639) |
none | new IOException("Error requesting access token", e) (line 643) |
UserCredentials |
refresh-token grant (line 297) | none | GoogleAuthException.createWithTokenEndpointIOException(e) (line 303) |
ExternalAccountCredentials |
STS exchange via StsRequestHandler (line 122) |
none | the raw IOException |
ExternalAccountCredentials with a service_account_impersonation_url refreshes through an internal ImpersonatedCredentials, so it inherits the impersonation row.
GoogleAuthException marks itself retryable, but nothing downstream reads that flag. google-cloud-core's BaseServiceException.isRetryable(boolean, IOException) and google-cloud-bigquery's default ExceptionHandler both match on the exception's own class (SocketException, ConnectException and so on). Neither the wrapped IOException nor GoogleAuthException matches.
Steps to reproduce
- Build an
ImpersonatedCredentialsorUserCredentialson anHttpTransportFactorywhose first request throwsjava.net.SocketException. - Call
refresh(). It fails on the first attempt. - Do the same with
ServiceAccountCredentials. It retries and succeeds.
Code example
AtomicInteger calls = new AtomicInteger();
MockHttpTransport dropsFirstRequest = new MockHttpTransport() {
@Override
public LowLevelHttpRequest buildRequest(String method, String url) {
return new MockLowLevelHttpRequest(url) {
@Override
public LowLevelHttpResponse execute() throws IOException {
if (calls.incrementAndGet() == 1) {
throw new SocketException("Unexpected end of file from server");
}
return new MockLowLevelHttpResponse()
.setContentType("application/json")
.setContent("{\"access_token\":\"t\",\"expires_in\":3600,\"token_type\":\"Bearer\"}");
}
};
}
};
UserCredentials user = UserCredentials.newBuilder()
.setClientId("c").setClientSecret("s").setRefreshToken("r")
.setHttpTransportFactory(() -> dropsFirstRequest)
.build();
user.refresh(); // throws GoogleAuthException; calls.get() == 1, no retryStack trace
From a BigQuery query authenticated with ImpersonatedCredentials in CI:
java.io.IOException: Error requesting access token
at com.google.auth.oauth2.ImpersonatedCredentials.refreshAccessToken(ImpersonatedCredentials.java)
...
Caused by: java.net.SocketException: Unexpected end of file from server
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java)
...
Suggested fix
Give the impersonation, user-credential and STS token requests the same HttpBackOffIOExceptionHandler that ServiceAccountCredentials already sets. All three requests can be repeated safely: minting a token has no side effects.
Additional information
We work around this in a BigQuery JDBC driver by wrapping each credential and retrying refreshAccessToken() when the cause chain shows a dropped connection: Two-Bear-Capital/tbc-bq-jdbc#356. With that in place, every credential type above recovers from a single dropped token request. A fix in the library would make the wrapper unnecessary.
Activity