apache / apache/datafusion-comet
Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs://
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
The native Iceberg write eligibility path supports `gs://` locations and `NativeConfig.extractObjectStoreOptions` extracts the corresponding `fs.gs.*` settings from the Hadoop configuration.
However, `CometIcebergNativeWrite` then passes those extracted settings through:
```scala
CometIcebergNativeScan.hadoopToIcebergS3Properties(...)
```
which translates S3/S3A configuration only.
For a `gs://` location, the extracted `fs.gs.*` settings are therefore not preserved when building the native GCS FileIO.
This is particularly problematic for tables using `HadoopFileIO`, because GCS credentials, endpoint configuration, and other filesystem settings may exist only in the Hadoop `Configuration` rather than `FileIO.properties()`.
The JVM Iceberg writer and the native writer can consequently operate with different effective storage configuration.
Depending on the environment, the native writer could:
* fail even though the JVM writer succeeds;
* use a different GCS endpoint; or
* resolve a different ambient/default identity than the one configured for the Hadoop filesystem.
Native writes should fail closed rather than silently changing storage identity or endpoint configuration.
### Steps to reproduce
_No response_
### Expected behavior
The native writer must either:
1. faithfully translate the relevant Hadoop GCS configuration into the native GCS backend configuration; or
2. fail closed and fall back to the JVM Iceberg writer for `HadoopFileIO + gs://`.
Until a complete and tested configuration bridge exists, the safer behavior is to decline native acceleration for this combination.
A table using an actual `GCSFileIO` whose properties already provide supported native `gcs.*` settings can remain eligible.
### Additional context
Found during follow-up review of #5361.
A simple `fs.gs.*` → `gcs.*` key rename is unlikely to be sufficient for all settings. For example, Hadoop GCS configuration may reference a service-account key file, whereas the native backend can require a different credential representation.
The important invariant for the write path is that enabling Comet must not cause the native writer to resolve a different storage identity or endpoint from the JVM writer.
Contributor guide
Research direction
Start with CometIcebergNativeWrite and NativeConfig.extractObjectStoreOptions, then inspect CometIcebergNativeScan.hadoopToIcebergS3Properties to trace how gs:// settings reach native GCS FileIO. Confirm the HadoopFileIO path preserves storage identity and endpoint configuration, or falls back to the JVM writer; validate the behavior with focused write-path tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, scala, spark
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100