HDFS support in iceberg-rust
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
There are several ways to access HDFS in rust:
- libhdfs + JNI: needs java runtime and libhdfs
- WebHDFS: http based, may have performance issues
- [HDFS Native](https://github.com/Kimahriman/hdfs-native): rust native but not well adopted.
OpenDAL supports all these methods. However, from Iceberg's perspective, it's better to expose a single `hdfs` storage option and allow users to choose the method via feature flags to avoid confusion. This is because we write service names in our metadata files, and all implementations must be able to read `hdfs://host:port/files`.
So my current plan is to introduce `webhdfs` and `hdfs_native` support in iceberg-rust. They will hide under `storage-webhdfs` and `storage-hdfs-native` flag:
Users can choose which implementations to use:
```rust
fn parse_scheme(scheme: &str) -> crate::Result {
match scheme {
"memory" => Ok(Scheme::Memory),
"file" | "" => Ok(Scheme::Fs),
"s3" | "s3a" => Ok(Scheme::S3),
"gs" | "gcs" => Ok(Scheme::Gcs),
#[cfg(feature = "storage-webhdfs")]
"hdfs" => Ok(Scheme::Webhdfs),
#[cfg(feature = "storage-hdfs-native")]
"hdfs" => Ok(Scheme::HdfsNative),
s => Ok(s.parse::()?),
}
}
```
See https://github.com/apache/iceberg-rust/pull/1131 for more details about this design.
Contributor guide
Research direction
Start by reading the proposed design in pull request 1131 and trace the parse_scheme entry point shown in this issue. Check how the storage-webhdfs and storage-hdfs-native feature flags select implementations, then verify that both support the hdfs://host:port/files service name used in metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100