apache / apache/arrow-rs

Expose public API for determining if Arrow IPC file is using compression

Open
#7,816 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

I'm writing a library for converting from Arrow IPC stream format to Arrow IPC file format, Parquet, and others. For the Arrow -> Arrow conversion I would love to be able to check that the file that has been written is using compression, and what compression it is using.

It's tricky since compression is per-buffer, and only applied if the compression actually saves space, but it would be nice to be able to access what compression was attempted. Maybe in metadata? I'd love thoughts on how this could work and if it's something that the project would be interested in having.

Thanks!!

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. First review the Arrow IPC file-writing and metadata APIs to determine where per-buffer compression attempts are represented. Done would be an agreed public API that reports whether compression was attempted and which compression was used, including buffers where compression did not save space.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.