apache / apache/hudi-rs

Support Hudi Variant reads via Arrow-rs

Open
#630 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
279
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
40

Description

## Background

Apache Hudi supports a Variant logical type represented in Avro as a record with `logicalType = "variant"` and encoded child fields such as `metadata`, `value`, and optionally `typed_value`. The hudi-rs read path is already Arrow-native, and arrow-rs/parquet-rs exposes Variant support through `parquet::variant::{VariantType, VariantArray}` behind the `parquet/variant_experimental` feature.

## Proposal

Add read compatibility for Hudi Variant columns by leaning on arrow-rs rather than introducing a separate hudi-rs Variant model.

## Scope

- Enable the parquet-rs Variant feature in the workspace dependency.
- Detect Hudi Avro records with `logicalType = "variant"` during Avro-to-Arrow schema conversion.
- Represent Variant columns as Arrow `Struct` fields carrying the `arrow.parquet.variant` extension metadata.
- Preserve existing read APIs that return Arrow `RecordBatch` values.
- Verify that returned columns can be converted with `parquet::variant::VariantArray::try_new`.
- Cover Parquet base-file reads and projection of Variant columns.

## Out of scope

- Hudi write support for Variant columns.
- New public hudi-rs-specific Variant APIs.
- SQL functions for querying nested Variant values.

## Validation

The implementation should pass targeted Avro/Parquet tests, workspace Rust clippy/format checks, all Rust workspace tests, and the MSRV workspace check.

Contributor guide

Open the contributing guide

Research direction

Start with the workspace parquet dependency and the Avro-to-Arrow schema conversion path. Add read handling for Hudi records with logicalType "variant", preserving Arrow RecordBatch APIs and the specified extension metadata. Add targeted Avro/Parquet tests for base-file reads, projection, and VariantArray::try_new conversion, then run the listed workspace checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.