apache / apache/datafusion-comet
Support Iceberg tables on HDFS in the native scan
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### What is the problem the feature request solves?
Comet's native Iceberg scan cannot read a table whose data lives on HDFS. `hdfs://` is not in the scheme allowlist that mirrors `storage_factory_for`, so any Iceberg table on HDFS falls back to the JVM reader and gets no native acceleration.
This is a gap relative to the plain-Parquet native scan, which does read HDFS (through libhdfs/JNI, via `fs.comet.libhdfs.schemes`). On-premise Iceberg deployments are commonly HDFS-backed, so the Iceberg acceleration Comet offers is unavailable to them today.
### Describe the potential solution
Route `hdfs://` to iceberg-rust's `hdfs-native` OpenDAL backend, a pure-Rust HDFS RPC client that needs no JNI or libhdfs. That backend is being added upstream in apache/iceberg-rust#3111; this issue tracks the Comet-side work once it lands.
Comet-side work:
- Add an `hdfs` arm to `storage_factory_for` and admit the scheme in the JVM gates (`CometScanRule.icebergReadableSchemes`, and `CometIcebergNativeWrite.SupportedStorageSchemes` for the write path).
- Forward the `hdfs.`/`hadoop.` property prefixes to the native `FileIO`.
- Resolve the NameNode endpoints for HA clusters. opendal's `HdfsNativeBuilder` never dials the authority written in the path: it builds one client against a synthetic authority and synthesizes the HA config from the comma-separated `hdfs.name-node` value. iceberg-rust falls back to the path authority only when that property is absent, which is correct just for a real `host:port`. An HA location reads `hdfs:///...`, and a nameservice is not a routable host, so the endpoints have to be derived from `dfs.ha.namenodes.` / `dfs.namenode.rpc-address..` in the session Hadoop configuration.
- Decline a scan whose data/delete files span more than one HDFS authority: one `hdfs.name-node` per scan overrides the authority of every path the FileIO opens, so a second nameservice would be read from the first one's NameNode at the same relative path.
### Additional context
Note that this is a second, independent HDFS client in the same process: the plain-Parquet path reaches HDFS through libhdfs/JNI while an Iceberg table would be opened over pure-Rust RPC. The two read the same `$HADOOP_CONF_DIR` XML but hold separate connections and separate Kerberos state.
Blocked on apache/iceberg-rust#3111.
Contributor guide
Research direction
The relevant entry points are storage_factory_for, CometScanRule.icebergReadableSchemes, and CometIcebergNativeWrite.SupportedStorageSchemes; first verify apache/iceberg-rust#3111 has landed. Trace the hdfs./hadoop. property forwarding and HA NameNode resolution from the session Hadoop configuration. Done means HDFS-backed Iceberg scans and writes use the native path, while scans spanning multiple HDFS authorities are declined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, rust
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100