Enhancement - provide option for gluten to pass s3 credentials in to velox using the configured spark.hadoop.fs.s3a.aws.credentials.provider
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Velox backend does not use the configured spark.hadoop.fs.s3a.aws.credentials.provider as its not using the Java AWS sdk, but rather the C++ one. One helpful option would be such that gluten can either eagerly resolve the credentials first using configured spark.hadoop.fs.s3a.aws.credentials.provider to init the velox backend.
Contributor guide
Research direction
Start by tracing Gluten's Velox backend initialization and how the configured spark.hadoop.fs.s3a.aws.credentials.provider is currently handled. Inspect the boundary between the Java AWS SDK credentials provider and the C++ Velox backend; done means the configured provider can supply credentials when initializing Velox.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, cpp, java
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100