apache / apache/hudi

[SUPPORT] Does Hudi support using Arrow IPC instead of parquet?

Open
#9,745 1 comment 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I got to know that arrow IPC files have significant performance benefits because the loading is much much faster and compression, while worse than parquet, is still pretty good.

The documentation states that hudi can be used with columnar formats like parquet and orc. Arrow IPC is such a format. I couldn't find any issue or documentation mentioning it.

Is it supported or planned to be supported?

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names Hudi documentation and the Parquet, ORC, and Arrow IPC formats but no source file, test, or entry point. First verify whether Arrow IPC is currently supported; if not, determine the affected data paths and define acceptance criteria for support or documentation.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.