apache / apache/datafusion-comet

Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs://

Open
#5,637 0 comments 0 reactions 0 assignees View on GitHub
bug requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

The native Iceberg write eligibility path supports `gs://` locations and `NativeConfig.extractObjectStoreOptions` extracts the corresponding `fs.gs.*` settings from the Hadoop configuration.

However, `CometIcebergNativeWrite` then passes those extracted settings through:

```scala
CometIcebergNativeScan.hadoopToIcebergS3Properties(...)
```

which translates S3/S3A configuration only.

For a `gs://` location, the extracted `fs.gs.*` settings are therefore not preserved when building the native GCS FileIO.

This is particularly problematic for tables using `HadoopFileIO`, because GCS credentials, endpoint configuration, and other filesystem settings may exist only in the Hadoop `Configuration` rather than `FileIO.properties()`.

The JVM Iceberg writer and the native writer can consequently operate with different effective storage configuration.

Depending on the environment, the native writer could:

* fail even though the JVM writer succeeds;
* use a different GCS endpoint; or
* resolve a different ambient/default identity than the one configured for the Hadoop filesystem.

Native writes should fail closed rather than silently changing storage identity or endpoint configuration.

### Steps to reproduce

_No response_

### Expected behavior

The native writer must either:

1. faithfully translate the relevant Hadoop GCS configuration into the native GCS backend configuration; or
2. fail closed and fall back to the JVM Iceberg writer for `HadoopFileIO + gs://`.

Until a complete and tested configuration bridge exists, the safer behavior is to decline native acceleration for this combination.

A table using an actual `GCSFileIO` whose properties already provide supported native `gcs.*` settings can remain eligible.

### Additional context

Found during follow-up review of #5361.

A simple `fs.gs.*` → `gcs.*` key rename is unlikely to be sufficient for all settings. For example, Hadoop GCS configuration may reference a service-account key file, whereas the native backend can require a different credential representation.

The important invariant for the write path is that enabling Comet must not cause the native writer to resolve a different storage identity or endpoint from the JVM writer.

Contributor guide

Open the contributing guide

Research direction

Start with CometIcebergNativeWrite and NativeConfig.extractObjectStoreOptions, then inspect CometIcebergNativeScan.hadoopToIcebergS3Properties to trace how gs:// settings reach native GCS FileIO. Confirm the HadoopFileIO path preserves storage identity and endpoint configuration, or falls back to the JVM writer; validate the behavior with focused write-path tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, scala, spark
Domain
backend, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.