apache / apache/arrow-rs

Runtime opt-in/-out for reading compressed IPC data

Open
#8,917 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
Record batches sent over IPC can be compressed (currently via LZ4 or ZSTD). Support for compressed can be enabled via [Cargo feature flags](https://github.com/apache/arrow-rs/blob/88b135a2fadfce80d7962ca0d91473f9c62337fb/arrow-ipc/Cargo.toml#L45-L46), but NOT at runtime.

Compression however is somewhat tricky, since decompressing untrusted data may easily spend many CPU cycles and result in large outputs (casually called "ZIP bombs"). Hence it would be nice to control -- from a readers PoV -- if compression is accepted or not.

**Describe the solution you'd like**
Add options to [`StreamReader`](https://docs.rs/arrow-ipc/latest/arrow_ipc/reader/struct.StreamReader.html) and [`FileReader`](https://docs.rs/arrow-ipc/latest/arrow_ipc/reader/struct.FileReader.html) to control if compression is accepted or not.

**Describe alternatives you've considered**
- **feature flags only:** Just rely on the feature flags currently exposed. This however is two downsides: a) feature-unification in a larger Cargo project may accidentally opt-in these flags and b) one single software artifact (e.g. a server communicating with an untrusted client and a trusted internal party in the same security domain) may have different requirements for different connections
- **expose decompressed size:** let the reader judge the impact of the compression by exposing the "size after decompression" / "uncompressed size". While this somewhat circumvents the unknown memory size after decompression, it only partly addresses the CPU overhead.

**Additional context**
- #7816 is somewhat related

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.