datafusion-contrib / datafusion-contrib/liquid-cache
To compression or not to compression?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 452
- Forks
- 51
- Avg merge
- 3h 10m
- Merged PRs (30d)
- 12
Description
We can send Arrow IPC with some lightweight compression (LZ4), but it seems that even with LZ4, the performance downgrade heavily when network bandwidth is sufficient.
But we also want compression when we run out of network bandwidth.
An approach is to hybrid: when we send a lot of data, try to compress; otherwise directly send it.
But hybrid approach is often a sign of bad taste. I don't want to do it until we have to do it.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue contains no file, test, or entry point to begin from. First clarify whether adaptive compression is wanted and define the bandwidth and performance thresholds; done would require an agreed approach and measurable behavior for choosing compression.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100