apache / apache/gluten

Enhancement - provide option for gluten to pass s3 credentials in to velox using the configured spark.hadoop.fs.s3a.aws.credentials.provider

Open
#6,012 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Velox backend does not use the configured spark.hadoop.fs.s3a.aws.credentials.provider as its not using the Java AWS sdk, but rather the C++ one. One helpful option would be such that gluten can either eagerly resolve the credentials first using configured spark.hadoop.fs.s3a.aws.credentials.provider to init the velox backend.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Gluten's Velox backend initialization and how the configured spark.hadoop.fs.s3a.aws.credentials.provider is currently handled. Inspect the boundary between the Java AWS SDK credentials provider and the C++ Velox backend; done means the configured provider can supply credentials when initializing Velox.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, cpp, java
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.