Feature Request: Support Clustering in hudi-rs
- Dominant language
- Rust
- Stars
- 279
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 40
Description
### Feature Description
Clustering is a core optimization feature in Apache Hudi, widely used to manage small files and improve query performance.
I’d love to see support for clustering in hudi-rs, which could handle this efficiently thanks to Rust’s performance. This would enable production-grade optimization workflows in native Rust pipelines.
### Why this matters
- Performance: Rust is expected to make compute-intensive operations like clustering much more performant
- Ease of migration: Users who are using standalone clustering should be able to migrate to hudi-rs clustering easily
- Ecosystem trend: Similar efforts are emerging in other formats, e.g. [Iceberg compaction](https://github.com/nimtable/iceberg-compaction/tree/main).
### Suggested Scope
Initial support might include:
- Reading clustering plans
- Executing clustering as a standalone action
- Supporting inline clustering in write paths (optional follow-up)
### Prerequisites
- hudi-rs doesn't have write support overall, it needs to be able to write and commit data before we implementing complicated table services
### Additional context
_No response_
Contributor guide
Research direction
Start by reviewing the existing write and commit support in hudi-rs, then trace how table services could read clustering plans and execute a standalone clustering action. The issue identifies inline clustering as an optional follow-up; done would require write and commit support plus an agreed initial clustering scope, but no files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100