Expose public API for determining if Arrow IPC file is using compression
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 1.3k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 167
Description
I'm writing a library for converting from Arrow IPC stream format to Arrow IPC file format, Parquet, and others. For the Arrow -> Arrow conversion I would love to be able to check that the file that has been written is using compression, and what compression it is using.
It's tricky since compression is per-buffer, and only applied if the compression actually saves space, but it would be nice to be able to access what compression was attempted. Maybe in metadata? I'd love thoughts on how this could work and if it's something that the project would be interested in having.
Thanks!!
Contributor guide
Research direction
The issue names no files, tests, or entry points. First review the Arrow IPC file-writing and metadata APIs to determine where per-buffer compression attempts are represented. Done would be an agreed public API that reports whether compression was attempted and which compression was used, including buffers where compression did not save space.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100