aws / aws/aws-sdk-java-v2

Provide a remedy for "Read timed out" causing failure to load credentials

Open
#3,051 1 comment 0 reactions 0 assignees View on GitHub
feature-request p2
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

### Describe the feature

Provide a way to configure the retry policy used for credentials requests, as well as a way to configure connect & read timeout values used for credentials requests.

Also, at minimum, throw a specific exception class (perhaps call it `CredentialsRetrievalException`) when credentials retrieval fails, so that users can easily add appropriate `catch` clauses & exception handling. That would be better than a generic `SdkClientException` which could probably be caused by many different problems, and which is a superclass of other more specific exception classes. For example, many database-related libraries, as well as HTTP response codes, differentiate between transient/retryable exceptions vs. non-transient/non-retryable exceptions. AWS SDK does have a `RetryableException` class which appears to serve a similar purpose, but it's not using it in this case even though it arguably should.

### Is your Feature Request related to a problem?

Currently, implementations of `ResourcesEndpointProvider` such as `InstanceProviderCredentialsEndpointProvider` specify a retryPolicy of `ResourcesEndpointRetryPolicy.NO_RETRY`, because they do not override what's provided by their `ResourcesEndpointProvider` interface. Therefore, a single failure to retrieve credentials will result in an exception thrown up to the caller of the AWS SDK.

Also, when making the HTTP calls, it uses the `ConnectionUtils` class which includes these hard-coded timeouts:

```
connection.setConnectTimeout(1000);
connection.setReadTimeout(1000);
```

There does not appear to be an easy way to change this behavior, without replacing the entire DefaultCredentialsProvider, at least, and possibly other things too.

Example stack trace (trimmed to AWS code only):

```
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:110)
at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
at software.amazon.awssdk.utils.cache.OneCallerBlocks.prefetch(OneCallerBlocks.java:38)
at software.amazon.awssdk.utils.cache.CachedSupplier.prefetchCache(CachedSupplier.java:116)
at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:91)
at java.util.Optional.map(Optional.java:215)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:85)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:105)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.resolveCredentials(AwsExecutionContextBuilder.java:165)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:102)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.batchWriteItem(DefaultDynamoDbClient.java:774)
..... (trimmed)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:743)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:116)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:79)
```

### Proposed Solution

_No response_

### Describe alternatives you've considered

Adding a catch block and retry logic similar to this:

```
} catch (SdkClientException e) {
if (e.getMessage() != null && e.getMessage().contains("Unable to load credentials")) {
LOGGER.warn("Operation XYZ failed during attempt {}.", attempts, e);
doXYZ(attempts + 1);
return;
} else {
throw e;
}
}
```

### Acknowledge

- [ ] I may be able to implement this feature request

### AWS Java SDK version used

2.17.36

### JDK version used

java version "1.8.0_221" Java(TM) SE Runtime Environment (build 1.8.0_221-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)

### Operating System and version

debian:jessie

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.