[SUPPORT] Does Hudi support using Arrow IPC instead of parquet?
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I got to know that arrow IPC files have significant performance benefits because the loading is much much faster and compression, while worse than parquet, is still pretty good.
The documentation states that hudi can be used with columnar formats like parquet and orc. Arrow IPC is such a format. I couldn't find any issue or documentation mentioning it.
Is it supported or planned to be supported?
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names Hudi documentation and the Parquet, ORC, and Arrow IPC formats but no source file, test, or entry point. First verify whether Arrow IPC is currently supported; if not, determine the affected data paths and define acceptance criteria for support or documentation.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100