apache / apache/iceberg-rust

HDFS support in iceberg-rust

Open
#1,130 28 comments 1 reaction 0 assignees View on GitHub
not-stale
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

There are several ways to access HDFS in rust:

- libhdfs + JNI: needs java runtime and libhdfs
- WebHDFS: http based, may have performance issues
- [HDFS Native](https://github.com/Kimahriman/hdfs-native): rust native but not well adopted.

OpenDAL supports all these methods. However, from Iceberg's perspective, it's better to expose a single `hdfs` storage option and allow users to choose the method via feature flags to avoid confusion. This is because we write service names in our metadata files, and all implementations must be able to read `hdfs://host:port/files`.

So my current plan is to introduce `webhdfs` and `hdfs_native` support in iceberg-rust. They will hide under `storage-webhdfs` and `storage-hdfs-native` flag:

Users can choose which implementations to use:

```rust
fn parse_scheme(scheme: &str) -> crate::Result {
match scheme {
"memory" => Ok(Scheme::Memory),
"file" | "" => Ok(Scheme::Fs),
"s3" | "s3a" => Ok(Scheme::S3),
"gs" | "gcs" => Ok(Scheme::Gcs),
#[cfg(feature = "storage-webhdfs")]
"hdfs" => Ok(Scheme::Webhdfs),
#[cfg(feature = "storage-hdfs-native")]
"hdfs" => Ok(Scheme::HdfsNative),
s => Ok(s.parse::()?),
}
}
```

See https://github.com/apache/iceberg-rust/pull/1131 for more details about this design.

Contributor guide

Open the contributing guide

Research direction

Start by reading the proposed design in pull request 1131 and trace the parse_scheme entry point shown in this issue. Check how the storage-webhdfs and storage-hdfs-native feature flags select implementations, then verify that both support the hdfs://host:port/files service name used in metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.