apache / apache/datafusion-comet

Support Iceberg tables on HDFS in the native scan

Open
#5,894 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Comet's native Iceberg scan cannot read a table whose data lives on HDFS. `hdfs://` is not in the scheme allowlist that mirrors `storage_factory_for`, so any Iceberg table on HDFS falls back to the JVM reader and gets no native acceleration.

This is a gap relative to the plain-Parquet native scan, which does read HDFS (through libhdfs/JNI, via `fs.comet.libhdfs.schemes`). On-premise Iceberg deployments are commonly HDFS-backed, so the Iceberg acceleration Comet offers is unavailable to them today.

### Describe the potential solution

Route `hdfs://` to iceberg-rust's `hdfs-native` OpenDAL backend, a pure-Rust HDFS RPC client that needs no JNI or libhdfs. That backend is being added upstream in apache/iceberg-rust#3111; this issue tracks the Comet-side work once it lands.

Comet-side work:

- Add an `hdfs` arm to `storage_factory_for` and admit the scheme in the JVM gates (`CometScanRule.icebergReadableSchemes`, and `CometIcebergNativeWrite.SupportedStorageSchemes` for the write path).
- Forward the `hdfs.`/`hadoop.` property prefixes to the native `FileIO`.
- Resolve the NameNode endpoints for HA clusters. opendal's `HdfsNativeBuilder` never dials the authority written in the path: it builds one client against a synthetic authority and synthesizes the HA config from the comma-separated `hdfs.name-node` value. iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a real `host:port`. An HA location reads `hdfs:///...`, and a nameservice is not a routable host, so the endpoints have to be derived from `dfs.ha.namenodes.` / `dfs.namenode.rpc-address..` in the session Hadoop configuration.
- Decline a scan whose data/delete files span more than one HDFS authority: one `hdfs.name-node` per scan overrides the authority of every path the FileIO opens, so a second nameservice would be read from the first one's NameNode at the same relative path.

### Additional context

Note that this is a second, independent HDFS client in the same process: the plain-Parquet path reaches HDFS through libhdfs/JNI while an Iceberg table would be opened over pure-Rust RPC. The two read the same `$HADOOP_CONF_DIR` XML but hold separate connections and separate Kerberos state.

Blocked on apache/iceberg-rust#3111.

Contributor guide

Open the contributing guide

Research direction

The relevant entry points are storage_factory_for, CometScanRule.icebergReadableSchemes, and CometIcebergNativeWrite.SupportedStorageSchemes; first verify apache/iceberg-rust#3111 has landed. Trace the hdfs./hadoop. property forwarding and HA NameNode resolution from the session Hadoop configuration. Done means HDFS-backed Iceberg scans and writes use the native path, while scans spanning multiple HDFS authorities are declined.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, rust
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.