apache / apache/fluss

[filesystem] Evaluate DefaultCredentialsProvider for S3 authentication

Open
#3,916 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Description

PR #3699 migrates the Fluss S3 filesystem plugin from AWS SDK v1 to AWS SDK v2. The current Glue guide documents separate `fs.s3a.aws.credentials.provider` lists for ECS Fargate, EKS with IRSA, and EC2.

As a follow-up to the review discussion in #3699, evaluate whether the guide can instead recommend the AWS SDK v2 `software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider` in `server.yaml`. This would let the AWS SDK own environment detection while still allowing users to override the provider for specialized setups.

### Verification

- [ ] Confirm that Hadoop S3A 3.4.3 can load `DefaultCredentialsProvider` through `fs.s3a.aws.credentials.provider`.
- [ ] Manually verify the configuration with ECS task-role credentials.
- [ ] Manually verify the configuration with EKS IRSA credentials.
- [ ] Manually verify the configuration with an EC2 instance profile.
- [ ] Verify both server-side `remote.data.dir` access and the relevant Fluss filesystem security-token/client access path.
- [ ] Check whether provider fallback introduces noticeable cold-start delays.
- [ ] If the verification succeeds, simplify the Glue guide and document how users can configure a specific provider when needed.

Related discussion: https://github.com/apache/fluss/pull/3699#issuecomment-5227873480
Related PR: #3699

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Glue guide and server.yaml, then check whether Hadoop S3A 3.4.3 loads software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider through fs.s3a.aws.credentials.provider. Verify ECS task-role, EKS IRSA, and EC2 instance-profile access for remote.data.dir and the relevant Fluss security-token/client path. Done means the approach is validated, fallback delays are understood, and the guide documents both the default and provider override configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
authentication, cloud, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.