apache / apache/polaris

Support credential vending for Spark Client (Delta tables)

Open
#3,020 3 comments 4 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Is your feature request related to a problem? Please describe.

When we use the Spark client for the delta table format support, we are having to configure spark with access credentials to access the underlying Storage bucket, as credential vending is not yet implemented

https://github.com/apache/polaris/blob/main/plugins/spark/v3.5/regtests/setup.sh#L165

Configuring static credentials defies the data governance goal that we use the Catalog for, therefore Credential Vending would unblock us for this case. Thanks in advance for all the support

### Describe the solution you'd like

Credential Vending for the Spark client (to support delta) similar to how vending is done for Iceberg

### Describe alternatives you've considered

An alternative is to configure Spark with the underlying S3 credentials (e.g. Access & Secret Key), but this is a security concern for our use case and we would like to avoid it

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with plugins/spark/v3.5/regtests/setup.sh around line 165 to understand the current static-credential setup. Compare the Spark client’s needs with how credential vending is done for Iceberg, then define the equivalent Delta-table flow and validate that Spark can access the underlying storage without configured access and secret keys.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend-api-design, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.