apache / apache/datafusion-comet

Scope object store credentials to execution payloads and selected authorities in the native Delta scan

Open
#5,659 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Follow-up from the native Delta Lake scan review (#5365). The planning payload for the native Delta scan retains object store options selected by provider-wide prefixes. Extracting credentials during execution payload construction, after final file selection, and including only the authorities required by the selected data files and DV locations would reduce credential retention and payload content.

Tests should verify that planning payloads contain no credentials, execution payloads contain only the selected scopes, and credentials for external DV locations remain available.

### Describe the potential solution

No response

### Additional context

Raised in the #5365 review discussion. No leak was demonstrated; this is contract hardening.

Contributor guide

Open the contributing guide

Research direction

Start with the native Delta scan planning and execution payload construction discussed in review #5365, then trace how object store options are selected for data files and deletion-vector locations. Add or update tests for credential-free planning payloads, execution payloads limited to selected authorities, and credentials retained for external DV locations.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala
Domain
data-engineering, security
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.