apache / apache/arrow

[C++] Improve compression strategy in IPC, Parquet

Open
#33,885 3 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Both Arrow IPC and the Parquet format allow optional compression of data buffers.
Currently, the heuristic used in the Arrow C++ codebase is simple: try to compress the entire data buffer, and write the compressed data if savings are achieved, otherwise write the uncompressed data (to save on decompression costs).

However, this heuristic always pays the full cost of compression even for uncompressible data (and compression is usually much more costly than decompression). This could be improved by employing a sampling strategy to reduce the cost of attempting to compress uncompressible data.

We could for example find inspiration in [Dask distributed's compression strategy](https://github.com/dask/distributed/blob/0063de53fed5e4e2e409940213c6265867e6635d/distributed/protocol/compression.py#L153).

### Component(s)

C++

Contributor guide

Open the contributing guide

Research direction

Start by locating the current compression heuristic in the Arrow C++ codebase for Arrow IPC and Parquet data buffers. Review Dask distributed's compression strategy for sampling ideas; done means reducing compression work for uncompressible data while still writing compressed data when sampling indicates meaningful savings.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.