apache / apache/datafusion-comet
Scope object store credentials to execution payloads and selected authorities in the native Delta scan
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Follow-up from the native Delta Lake scan review (#5365). The planning payload for the native Delta scan retains object store options selected by provider-wide prefixes. Extracting credentials during execution payload construction, after final file selection, and including only the authorities required by the selected data files and DV locations would reduce credential retention and payload content.
Tests should verify that planning payloads contain no credentials, execution payloads contain only the selected scopes, and credentials for external DV locations remain available.
### Describe the potential solution
No response
### Additional context
Raised in the #5365 review discussion. No leak was demonstrated; this is contract hardening.
Contributor guide
Research direction
Start with the native Delta scan planning and execution payload construction discussed in review #5365, then trace how object store options are selected for data files and deletion-vector locations. Add or update tests for credential-free planning payloads, execution payloads limited to selected authorities, and credentials retained for external DV locations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala
- Domain
- data-engineering, security
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100