apache / apache/datafusion-comet

Regular native scan forwards S3 options without checking divergence from Hadoop's resolution

Open
#5,662 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

Raised by @viirya in the #5365 review. `NativeConfig.extractObjectStoreOptions` forwards `fs.s3a.*` options to the native object store client with no divergence check, but Comet's native client resolves S3 configuration differently from Hadoop's S3AFileSystem in several ways (lookupPassword bucket precedence, JCEKS credential provider aliases, clear text fallback, assumed role session policies, provider class semantics). A plain Parquet table read through the regular native scan can therefore read under a different effective identity or endpoint than Spark would, silently.

The Delta contrib in #5365 carries a divergence comparator that detects these cases and declines to Spark, but the regular native scan path has no equivalent gate. #5658 tracks extracting that comparator into a shared helper; this issue tracks actually applying it (or an equivalent check) to the regular native scan so the gap is closed for every native read, not only Delta.

### Steps to reproduce

Configure any setting the two resolvers treat differently, for example a JCEKS-backed fs.s3a.secret.key with hadoop.security.credential.clear-text-fallback=false plus a clear text value, and read a plain Parquet table with the native scan enabled.

### Expected behavior

The scan declines to Spark when native would resolve different S3 configuration than Hadoop, or resolves identically.

### Additional context

No response

Contributor guide

Open the contributing guide

Research direction

Start at NativeConfig.extractObjectStoreOptions and read the divergence comparator described in #5365 and #5658, then trace where the regular native scan decides whether to run natively or decline to Spark. Reproduce the JCEKS and clear-text fallback case against a plain Parquet table; done means divergent S3 resolution declines to Spark while equivalent resolution remains native.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, hadoop, scala, spark
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.