aws / aws/aws-sdk-java-v2

Blocking call while resolving credentials inside an EKS cluster using DynamoDB async client (Blockhound)

Open
#2,360 2 comments 1 reaction 0 assignees View on GitHub
bug p2
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

## Describe the bug
While testing a spring boot webflux app inside a EKS cluster I've detected a blocking call within AWS libraries thanks to [BlockHound](https://github.com/reactor/BlockHound).

It happens while resolving the credentials right before the dynamodb call using the async client. This happens even when I try to provide a custom aws credentials provider:

```
@Bean
fun awsCredentialsProviderChain(): AwsCredentialsProviderChain {
return AwsCredentialsProviderChain.of(
ContainerCredentialsProvider.builder().asyncCredentialUpdateEnabled(true)
.build(),
InstanceProfileCredentialsProvider.builder()
.asyncCredentialUpdateEnabled(true)
.build()
)
}
```

My dynamodb client:
```

@Bean
fun dynamoDbAsyncClient(awsCredentialsProviderChain: AwsCredentialsProviderChain): DynamoDbAsyncClient =
DynamoDbAsyncClient
.builder().credentialsProvider(awsCredentialsProviderChain).build()
```

Locally everything works fine, I think the issue lies inside the _InstanceProfileCredentialsProvider_, the other providers seem to be working fine.

## Expected Behavior
Dynamodb aync client should be non blocking so I'd expect to use it without any blocking cals

## Current Behavior

Blockhounds detects a blocking call.
Stacktrace:

`reactor.blockhound.BlockingOperationError: Blocking call! java.net.SocketOutputStream#socketWrite0
at java.base/java.net.SocketOutputStream.socketWrite0(Unknown Source)
Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
|_ checkpoint ⇢ com.mypackage.MyFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
|_ checkpoint ⇢ HTTP GET "/v1/trolleys/1001781417" [ExceptionHandlingWebHandler]
Stack trace:
at java.base/java.net.SocketOutputStream.socketWrite0(Unknown Source)
at java.base/java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.base/java.net.SocketOutputStream.write(Unknown Source)
at java.base/java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.base/java.io.BufferedOutputStream.flush(Unknown Source)
at java.base/java.io.PrintStream.flush(Unknown Source)
at java.base/sun.net.www.MessageHeader.print(Unknown Source)
at java.base/sun.net.www.http.HttpClient.writeRequests(Unknown Source)
at java.base/sun.net.www.http.HttpClient.writeRequests(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.writeRequests(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
at software.amazon.awssdk.regions.util.HttpResourcesUtils.readResource(HttpResourcesUtils.java:114)
at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.getToken(EC2MetadataUtils.java:442)
at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getToken(InstanceProfileCredentialsProvider.java:83)
at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getCredentialsEndpointProvider(InstanceProfileCredentialsProvider.java:69)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:74)
at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:89)
at java.base/java.util.Optional.map(Unknown Source)
at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146)
at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:91)
at software.amazon.awssdk.awscore.client.handler.AwsClientHandlerUtils.createExecutionContext(AwsClientHandlerUtils.java:79)
at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.createExecutionContext(AwsAsyncClientHandler.java:65)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:78)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:276)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:75)
at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:52)
at software.amazon.awssdk.services.dynamodb.DefaultDynamoDbAsyncClient.getItem(DefaultDynamoDbAsyncClient.java:3256)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.CommonOperation.executeAsync(CommonOperation.java:140)
at software.amazon.awssdk.enhanced.dynamodb.internal.operations.TableOperation.executeOnPrimaryIndexAsync(TableOperation.java:81)
at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbAsyncTable.getItem(DefaultDynamoDbAsyncTable.java:136)
at software.amazon.awssdk.enhanced.dynamodb.internal.client.DefaultDynamoDbAsyncTable.getItem(DefaultDynamoDbAsyncTable.java:143)`

## Steps to Reproduce
Run a spring boot webflux app with a dynamodb async client with blockound installed inside an EKS cluster. Using the client will throw an exception

## Context
Not sure if this is going to have any impact on the service perfomance. The main point here is that this issue prevents the client of being full non blocking.

Meanwhile, I can ignore this blocking call on the blockhound integration configurations.

## Your Environment
* AWS Java SDK version used: 2.16.25
* JDK version used: openjdk-11-jre-ubi-minimal:11.0.10-hotspot
* Kotlin version: 1.4.31
* Spring boot version: 2.4.4

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.