Provide a remedy for "Read timed out" causing failure to load credentials
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 51
Description
### Describe the feature
Provide a way to configure the retry policy used for credentials requests, as well as a way to configure connect & read timeout values used for credentials requests.
Also, at minimum, throw a specific exception class (perhaps call it `CredentialsRetrievalException`) when credentials retrieval fails, so that users can easily add appropriate `catch` clauses & exception handling. That would be better than a generic `SdkClientException` which could probably be caused by many different problems, and which is a superclass of other more specific exception classes. For example, many database-related libraries, as well as HTTP response codes, differentiate between transient/retryable exceptions vs. non-transient/non-retryable exceptions. AWS SDK does have a `RetryableException` class which appears to serve a similar purpose, but it's not using it in this case even though it arguably should.
### Is your Feature Request related to a problem?
Currently, implementations of `ResourcesEndpointProvider` such as `InstanceProviderCredentialsEndpointProvider` specify a retryPolicy of `ResourcesEndpointRetryPolicy.NO_RETRY`, because they do not override what's provided by their `ResourcesEndpointProvider` interface. Therefore, a single failure to retrieve credentials will result in an exception thrown up to the caller of the AWS SDK.
Also, when making the HTTP calls, it uses the `ConnectionUtils` class which includes these hard-coded timeouts:
```
connection.setConnectTimeout(1000);
connection.setReadTimeout(1000);
```
There does not appear to be an easy way to change this behavior, without replacing the entire DefaultCredentialsProvider, at least, and possibly other things too.
Example stack trace (trimmed to AWS code only):
```
software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from service endpoint.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:98)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:110)
at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
at software.amazon.awssdk.utils.cache.OneCallerBlocks.prefetch(OneCallerBlocks.java:38)
at software.amazon.awssdk.utils.cache.CachedSupplier.prefetchCache(CachedSupplier.java:116)
at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:91)
at java.util.Optional.map(Optional.java:215)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:85)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:105)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.resolveCredentials(AwsExecutionContextBuilder.java:165)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:102)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:175)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56)
at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbClient.batchWriteItem(DefaultDynamoDbClient.java:774)
..... (trimmed)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:743)
at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:116)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:91)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:79)
```
### Proposed Solution
_No response_
### Describe alternatives you've considered
Adding a catch block and retry logic similar to this:
```
} catch (SdkClientException e) {
if (e.getMessage() != null && e.getMessage().contains("Unable to load credentials")) {
LOGGER.warn("Operation XYZ failed during attempt {}.", attempts, e);
doXYZ(attempts + 1);
return;
} else {
throw e;
}
}
```
### Acknowledge
- [ ] I may be able to implement this feature request
### AWS Java SDK version used
2.17.36
### JDK version used
java version "1.8.0_221" Java(TM) SE Runtime Environment (build 1.8.0_221-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.221-b11, mixed mode)
### Operating System and version
debian:jessie
Contributor guide
Assessment
This issue has not been assessed yet.