[C++] Improve compression strategy in IPC, Parquet
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Both Arrow IPC and the Parquet format allow optional compression of data buffers.
Currently, the heuristic used in the Arrow C++ codebase is simple: try to compress the entire data buffer, and write the compressed data if savings are achieved, otherwise write the uncompressed data (to save on decompression costs).
However, this heuristic always pays the full cost of compression even for uncompressible data (and compression is usually much more costly than decompression). This could be improved by employing a sampling strategy to reduce the cost of attempting to compress uncompressible data.
We could for example find inspiration in [Dask distributed's compression strategy](https://github.com/dask/distributed/blob/0063de53fed5e4e2e409940213c6265867e6635d/distributed/protocol/compression.py#L153).
### Component(s)
C++
Contributor guide
Research direction
Start by locating the current compression heuristic in the Arrow C++ codebase for Arrow IPC and Parquet data buffers. Review Dask distributed's compression strategy for sampling ideas; done means reducing compression work for uncompressible data while still writing compressed data when sampling indicates meaningful savings.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100