apache / apache/arrow-rs

Add snappy_cpp as optional dependency

Open
#9,530 2 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 18h
Merged PRs (30d)
169

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

It seems much faster than `snap` in my testing on compressing, even on e2e benchmarks (clickbench) it is up to 2x as fast:

```
arrow_reader_clickbench/async_object_store/Q1
time: [906.71 µs 910.64 µs 914.99 µs]
change: [−3.1264% −1.9584% −0.9019%] (p = 0.00 < 0.05)
Change within noise threshold.
Found 10 outliers among 100 measurements (10.00%)
7 (7.00%) high mild
3 (3.00%) high severe
arrow_reader_clickbench/async_object_store/Q10
time: [5.4424 ms 5.4692 ms 5.4999 ms]
change: [−2.5400% −1.0993% +0.0171%] (p = 0.09 > 0.05)
No change in performance detected.
Found 16 outliers among 100 measurements (16.00%)
3 (3.00%) high mild
13 (13.00%) high severe
arrow_reader_clickbench/async_object_store/Q11
time: [6.1418 ms 6.1514 ms 6.1619 ms]
change: [−0.4752% +0.1639% +0.7037%] (p = 0.60 > 0.05)
No change in performance detected.
Found 10 outliers among 100 measurements (10.00%)
7 (7.00%) high mild
3 (3.00%) high severe
arrow_reader_clickbench/async_object_store/Q12
time: [11.065 ms 11.089 ms 11.125 ms]
change: [−17.577% −17.329% −17.043%] (p = 0.00 < 0.05)
Performance has improved.
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe
arrow_reader_clickbench/async_object_store/Q13
time: [13.540 ms 13.578 ms 13.620 ms]
change: [−13.550% −13.282% −12.955%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe
arrow_reader_clickbench/async_object_store/Q14
time: [12.368 ms 12.481 ms 12.643 ms]
change: [−16.844% −16.059% −14.979%] (p = 0.00 < 0.05)
Performance has improved.
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe
arrow_reader_clickbench/async_object_store/Q19
time: [2.3651 ms 2.3716 ms 2.3784 ms]
change: [−0.2989% +0.4660% +1.1652%] (p = 0.23 > 0.05)
No change in performance detected.
arrow_reader_clickbench/async_object_store/Q20
time: [46.271 ms 46.409 ms 46.592 ms]
change: [−37.543% −37.315% −37.054%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) high mild
3 (3.00%) high severe
Benchmarking arrow_reader_clickbench/async_object_store/Q21: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 5.3s, or reduce sample count to 90.
arrow_reader_clickbench/async_object_store/Q21
time: [53.187 ms 53.314 ms 53.444 ms]
change: [−35.829% −35.618% −35.418%] (p = 0.00 < 0.05)
Performance has improved.
Benchmarking arrow_reader_clickbench/async_object_store/Q22: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 6.7s, or reduce sample count to 70.
arrow_reader_clickbench/async_object_store/Q22
time: [67.068 ms 67.386 ms 67.793 ms]
change: [−32.873% −32.418% −31.971%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
5 (5.00%) high mild
3 (3.00%) high severe
Benchmarking arrow_reader_clickbench/async_object_store/Q23: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 5.0s. You may wish to increase target time to 15.4s, or reduce sample count to 30.
arrow_reader_clickbench/async_object_store/Q23
time: [147.95 ms 148.29 ms 148.67 ms]
change: [−33.220% −33.030% −32.818%] (p = 0.00 < 0.05)
Performance has improved.
Found 5 outliers among 100 measurements (5.00%)
4 (4.00%) high mild
1 (1.00%) high severe
arrow_reader_clickbench/async_object_store/Q24
time: [15.013 ms 15.030 ms 15.048 ms]
change: [−16.445% −16.294% −16.150%] (p = 0.00 < 0.05)
Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
1 (1.00%) low mild
5 (5.00%) high mild
2 (2.00%) high severe
arrow_reader_clickbench/async_object_store/Q27
time: [32.775 ms 32.842 ms 32.922 ms]
change: [−46.691% −46.261% −45.930%] (p = 0.00 < 0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
1 (1.00%) high mild
3 (3.00%) high severe
arrow_reader_clickbench/async_object_store/Q28
time: [35.060 ms 35.184 ms 35.342 ms]
change: [−42.107% −41.869% −41.569%] (p = 0.00 < 0.05)
Performance has improved.
Found 9 outliers among 100 measurements (9.00%)
3 (3.00%) high mild
6 (6.00%) high severe
arrow_reader_clickbench/async_object_store/Q30
time: [14.191 ms 14.216 ms 14.245 ms]
change: [−15.213% −15.022% −14.818%] (p = 0.00 < 0.05)
Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
```

**Describe the solution you'd like**
Add it

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start by locating the Rust dependency and feature declarations for compression support; the issue does not name specific files or tests. Review how the existing snap dependency is selected, then use the arrow_reader_clickbench/async_object_store benchmarks mentioned in the report to compare the optional configuration. Done means snappy_cpp can be enabled as an optional dependency without affecting the default setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, rust
Domain
build-system, data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.