Support Hudi Variant reads via Arrow-rs
- Dominant language
- Rust
- Stars
- 279
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 40
Description
## Background
Apache Hudi supports a Variant logical type represented in Avro as a record with `logicalType = "variant"` and encoded child fields such as `metadata`, `value`, and optionally `typed_value`. The hudi-rs read path is already Arrow-native, and arrow-rs/parquet-rs exposes Variant support through `parquet::variant::{VariantType, VariantArray}` behind the `parquet/variant_experimental` feature.
## Proposal
Add read compatibility for Hudi Variant columns by leaning on arrow-rs rather than introducing a separate hudi-rs Variant model.
## Scope
- Enable the parquet-rs Variant feature in the workspace dependency.
- Detect Hudi Avro records with `logicalType = "variant"` during Avro-to-Arrow schema conversion.
- Represent Variant columns as Arrow `Struct` fields carrying the `arrow.parquet.variant` extension metadata.
- Preserve existing read APIs that return Arrow `RecordBatch` values.
- Verify that returned columns can be converted with `parquet::variant::VariantArray::try_new`.
- Cover Parquet base-file reads and projection of Variant columns.
## Out of scope
- Hudi write support for Variant columns.
- New public hudi-rs-specific Variant APIs.
- SQL functions for querying nested Variant values.
## Validation
The implementation should pass targeted Avro/Parquet tests, workspace Rust clippy/format checks, all Rust workspace tests, and the MSRV workspace check.
Contributor guide
Research direction
Start with the workspace parquet dependency and the Avro-to-Arrow schema conversion path. Add read handling for Hudi records with logicalType "variant", preserving Arrow RecordBatch APIs and the specified extension metadata. Add targeted Avro/Parquet tests for base-file reads, projection, and VariantArray::try_new conversion, then run the listed workspace checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100