apache / apache/datafusion-comet
Regular native scan forwards S3 options without checking divergence from Hadoop's resolution
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
Raised by @viirya in the #5365 review. `NativeConfig.extractObjectStoreOptions` forwards `fs.s3a.*` options to the native object store client with no divergence check, but Comet's native client resolves S3 configuration differently from Hadoop's S3AFileSystem in several ways (lookupPassword bucket precedence, JCEKS credential provider aliases, clear text fallback, assumed role session policies, provider class semantics). A plain Parquet table read through the regular native scan can therefore read under a different effective identity or endpoint than Spark would, silently.
The Delta contrib in #5365 carries a divergence comparator that detects these cases and declines to Spark, but the regular native scan path has no equivalent gate. #5658 tracks extracting that comparator into a shared helper; this issue tracks actually applying it (or an equivalent check) to the regular native scan so the gap is closed for every native read, not only Delta.
### Steps to reproduce
Configure any setting the two resolvers treat differently, for example a JCEKS-backed fs.s3a.secret.key with hadoop.security.credential.clear-text-fallback=false plus a clear text value, and read a plain Parquet table with the native scan enabled.
### Expected behavior
The scan declines to Spark when native would resolve different S3 configuration than Hadoop, or resolves identically.
### Additional context
No response
Contributor guide
Research direction
Start at NativeConfig.extractObjectStoreOptions and read the divergence comparator described in #5365 and #5658, then trace where the regular native scan decides whether to run natively or decline to Spark. Reproduce the JCEKS and clear-text fallback case against a plain Parquet table; done means divergent S3 resolution declines to Spark while equivalent resolution remains native.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, hadoop, scala, spark
- Domain
- backend, cloud, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100