aws / aws/aws-sdk-java-v2

"Unable to load credentials from any of the providers" for parallel calls to an S3Client with ProfileCredentialsProvider

Open
#6,827 2 comments 0 reactions 1 assignee Claimed by @debora-ito View on GitHub
bug p2
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

### Describe the bug

I am still encountering the behavior from this closed issue https://github.com/aws/aws-sdk-java-v2/issues/5001.

In my case I have a list of S3-Objects that exist in a bucket and download & process them in a parallel stream.

It sometimes manages some objects, but **always** fails with `Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain...` (see stacktrace) eventually. The number of elements doesn't seem to have an impact.

In my local testing the same program worked flawlessly when either the stream was sequential instead of parallel or the S3Client has already executed another call before the parallel stream starts.

This makes the analysis from that previous issue seem likely, that there's a race condition for `resolveCredentials` in ProfileCredentialProvider.

### Regression Issue

- [ ] Select this option if this issue appears to be a regression.

### Expected Behavior

The parallel stream should be processed normally.

### Current Behavior

> Exception in thread "main" software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=_**my profile name**_, profileFile=ProfileFile(sections=[profiles, sso-session, services], profiles=[Profile(name=default, properties=[region, output, ca_bundle]), Profile(name=_**my profile name**_, properties=[role_arn, region, source_profile])])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=_**my profile name**_): Cannot invoke "software.amazon.awssdk.profiles.ProfileFile.profile(String)" because "profileFile" is null, ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:130)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130)
at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:134)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.lambda$trySelectAuthScheme$6(S3AuthSchemeInterceptor.java:181)
at software.amazon.awssdk.core.internal.util.MetricUtils.reportDuration(MetricUtils.java:81)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.trySelectAuthScheme(S3AuthSchemeInterceptor.java:181)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.selectAuthScheme(S3AuthSchemeInterceptor.java:99)
at software.amazon.awssdk.services.s3.auth.scheme.internal.S3AuthSchemeInterceptor.beforeExecution(S3AuthSchemeInterceptor.java:71)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$beforeExecution$1(ExecutionInterceptorChain.java:59)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.beforeExecution(ExecutionInterceptorChain.java:59)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.runInitialInterceptors(AwsExecutionContextBuilder.java:319)
at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:155)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:67)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$0(BaseSyncClientHandler.java:62)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:60)
at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:52)
at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:60)
at software.amazon.awssdk.services.s3.DefaultS3Client.getObject(DefaultS3Client.java:6837)
at software.amazon.awssdk.services.s3.S3Client.getObject(S3Client.java:11977)
at software.amazon.awssdk.services.s3.S3Client.getObject(S3Client.java:12243)
at Test.lambda$main$1(Test.java:89)

### Reproduction Steps

A simple version of my program could look like this:
```
try (S3Client s3Client = S3Client.builder().region(Region.EU_CENTRAL_1).credentialsProvider(DefaultCredentialsProvider.builder().build()).build()) {
// works if s3Client is already called here with e.g:
// s3Client.listBuckets().buckets().forEach(IO::println);

String bucketName = "...";
List s3Keys = List.of("...", "...", "...", ...);
s3Keys.parallelStream() // works if this is just .stream()
.map(s3Key -> {
try {
return new String(s3Client.getObject(b -> b.bucket(bucketName).key(s3Key)).readAllBytes());
} catch (IOException e) {
throw new RuntimeException(e);
}
})
.map(String::length)
.forEach(System.out::println);
}
```

### Possible Solution

_No response_

### Additional Information/Context

_No response_

### AWS Java SDK version used

2.42.4

### JDK version used

amazon-corretto-jdk25.0.0_36

### Operating System and version

Windows 11

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.