apache / apache/iceberg

AWS: client credentials provider NPE

Open Beginner friendly
#17,536 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

None

### Please describe the bug 🐞

As of 1.11.0 (https://github.com/apache/iceberg/pull/15249), `AwsClientProperties#credentialsProvider` unconditionally does a `clientCredentialsProviderProperties.put(VendedCredentialsProvider.URI, refreshCredentialsEndpoint)` if "client.credentials-provider" is set; this may result in `null` being put into the properties map. Other similar code paths check first to make sure the value is not `null` or empty before putting the value.

This does not play well with custom credentials providers / `org.apache.iceberg.aws.AwsClientProperties`:

```
java.lang.NullPointerException
at java.base/java.util.Objects.requireNonNull(Objects.java:233)
at java.base/java.util.stream.Collectors.lambda$uniqKeysMapAccumulator$1(Collectors.java:180)
at java.base/java.util.stream.ReduceOps$3ReducingSink.accept(ReduceOps.java:169)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:179)
at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1858)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
at org.apache.iceberg.util.PropertyUtil.filterProperties(PropertyUtil.java:191)
at org.apache.iceberg.aws.AwsClientProperties.(AwsClientProperties.java:116)
at io.deephaven.iceberg.util.DeephavenS3ClientCredentialsProvider.create(DeephavenS3ClientCredentialsProvider.java:29)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:60)
at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:73)
at org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:186)
at org.apache.iceberg.aws.AwsClientProperties.createCredentialsProvider(AwsClientProperties.java:314)
at org.apache.iceberg.aws.AwsClientProperties.credentialsProvider(AwsClientProperties.java:296)
at org.apache.iceberg.aws.AwsClientProperties.credentialsProvider(AwsClientProperties.java:218)
at org.apache.iceberg.aws.s3.S3FileIOProperties.getCredentialsProvider(S3FileIOProperties.java:996)
at org.apache.iceberg.aws.s3.S3FileIOProperties.applyCredentialConfigurations(S3FileIOProperties.java:985)
at org.apache.iceberg.aws.AwsClientFactories$DefaultAwsClientFactory.lambda$s3$0(AwsClientFactories.java:116)
at software.amazon.awssdk.utils.builder.SdkBuilder.applyMutation(SdkBuilder.java:61)
at org.apache.iceberg.aws.AwsClientFactories$DefaultAwsClientFactory.s3(AwsClientFactories.java:115)
at io.deephaven.iceberg.util.DeephavenAwsClientFactory.s3(DeephavenAwsClientFactory.java:49)
```

I'm unclear the exact semantics of nulls in property maps, and if this should be considered an upstream issue (ie, `null` should not be put in the map), or a downstream issue (`AwsClientProperties` should be tolerant of entries that have `null` values).

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start in org.apache.iceberg.aws.AwsClientProperties#credentialsProvider and inspect the clientCredentialsProviderProperties handling before PropertyUtil.filterProperties is called. Reproduce the custom credentials-provider case with no refresh endpoint, then verify the AWS client can be created without the reported NullPointerException and add coverage in the relevant AWS properties tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
authentication, cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.