Support credential vending for Spark Client (Delta tables)
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Is your feature request related to a problem? Please describe.
When we use the Spark client for the delta table format support, we are having to configure spark with access credentials to access the underlying Storage bucket, as credential vending is not yet implemented
https://github.com/apache/polaris/blob/main/plugins/spark/v3.5/regtests/setup.sh#L165
Configuring static credentials defies the data governance goal that we use the Catalog for, therefore Credential Vending would unblock us for this case. Thanks in advance for all the support
### Describe the solution you'd like
Credential Vending for the Spark client (to support delta) similar to how vending is done for Iceberg
### Describe alternatives you've considered
An alternative is to configure Spark with the underlying S3 credentials (e.g. Access & Secret Key), but this is a security concern for our use case and we would like to avoid it
### Additional context
_No response_
Contributor guide
Research direction
Start with plugins/spark/v3.5/regtests/setup.sh around line 165 to understand the current static-credential setup. Compare the Spark client’s needs with how credential vending is done for Iceberg, then define the equivalent Delta-table flow and validate that Spark can access the underlying storage without configured access and secret keys.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend-api-design, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100