apache / apache/datafusion-comet

Move cloud compatibility gating into a shared helper beside the object store configuration code

Open
#5,658 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

### What is the problem the feature request solves?

Follow-up from the native Delta Lake scan review (#5365). The Delta contrib carries its own cloud compatibility gating (S3 option divergence checks) next to Delta admission logic. Since native Parquet and native Delta share the same underlying object store client, that policy could live in a focused helper beside the shared object store configuration code so compatibility fixes apply consistently, and Delta admission can concentrate on protocol features, schemas, and Delta metadata.

The helper must preserve Hadoop's distinct rules for plain options, password-backed values, variable substitution, and bucket overrides, without introducing another provider abstraction.

### Describe the potential solution

No response

### Additional context

Raised in the #5365 review discussion.

Contributor guide

Open the contributing guide

Research direction

Start with the native Parquet and native Delta object-store configuration and Delta admission paths referenced in the #5365 review. Trace the existing S3 option-divergence checks and compare Hadoop's rules for plain options, password-backed values, variable substitution, and bucket overrides. Done means a shared helper beside the object-store configuration is used consistently, while Delta admission focuses on protocol features, schemas, and metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, scala
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.