apache / apache/datafusion-comet
Move cloud compatibility gating into a shared helper beside the object store configuration code
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 190
Description
### What is the problem the feature request solves?
Follow-up from the native Delta Lake scan review (#5365). The Delta contrib carries its own cloud compatibility gating (S3 option divergence checks) next to Delta admission logic. Since native Parquet and native Delta share the same underlying object store client, that policy could live in a focused helper beside the shared object store configuration code so compatibility fixes apply consistently, and Delta admission can concentrate on protocol features, schemas, and Delta metadata.
The helper must preserve Hadoop's distinct rules for plain options, password-backed values, variable substitution, and bucket overrides, without introducing another provider abstraction.
### Describe the potential solution
No response
### Additional context
Raised in the #5365 review discussion.
Contributor guide
Research direction
Start with the native Parquet and native Delta object-store configuration and Delta admission paths referenced in the #5365 review. Trace the existing S3 option-divergence checks and compare Hadoop's rules for plain options, password-backed values, variable substitution, and bucket overrides. Done means a shared helper beside the object-store configuration is used consistently, while Delta admission focuses on protocol features, schemas, and metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, scala
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100