BurntSushi / BurntSushi/critcmp
human sort the benchmarks
- Dominant language
- Rust
- Stars
- 271
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
We have a set of micro benchmarks which we compare using `critcmp`. This would be an example benchmark result:
```console
group benchmark-current.baseline benchmark-main.baseline
----- -------------------------- -----------------------
fibonacci_loop/our/1 1.06 196.4±0.37ns 4.9 MElem/sec 1.00 185.8±4.38ns 5.1 MElem/sec
fibonacci_loop/our/1024 1.01 78.3±0.07µs 12.5 MElem/sec 1.00 77.5±0.09µs 12.6 MElem/sec
fibonacci_loop/our/1048576 1.01 80.5±0.27ms 12.4 MElem/sec 1.00 79.9±0.21ms 12.5 MElem/sec
fibonacci_loop/our/128 1.00 9.9±0.05µs 12.4 MElem/sec 1.00 9.8±0.01µs 12.4 MElem/sec
fibonacci_loop/our/131072 1.00 10.0±0.01ms 12.6 MElem/sec 1.01 10.1±0.01ms 12.4 MElem/sec
fibonacci_loop/our/16 1.00 1346.4±1.96ns 11.3 MElem/sec 1.00 1352.6±1.23ns 11.3 MElem/sec
fibonacci_loop/our/16384 1.01 1246.7±1.13µs 12.5 MElem/sec 1.00 1237.5±1.13µs 12.6 MElem/sec
fibonacci_loop/our/2 1.04 282.9±1.63ns 6.7 MElem/sec 1.00 271.3±2.37ns 7.0 MElem/sec
fibonacci_loop/our/2048 1.02 158.5±0.15µs 12.3 MElem/sec 1.00 154.8±0.14µs 12.6 MElem/sec
fibonacci_loop/our/256 1.01 19.6±0.02µs 12.4 MElem/sec 1.00 19.5±0.03µs 12.5 MElem/sec
fibonacci_loop/our/262144 1.00 19.9±0.04ms 12.5 MElem/sec 1.01 20.1±0.02ms 12.5 MElem/sec
fibonacci_loop/our/32 1.01 2.6±0.00µs 11.9 MElem/sec 1.00 2.5±0.00µs 12.0 MElem/sec
fibonacci_loop/our/32768 1.01 2.5±0.00ms 12.5 MElem/sec 1.00 2.5±0.00ms 12.6 MElem/sec
fibonacci_loop/our/4 1.01 433.6±0.98ns 8.8 MElem/sec 1.00 428.4±0.66ns 8.9 MElem/sec
fibonacci_loop/our/4096 1.04 321.8±36.30µs 12.1 MElem/sec 1.00 309.5±0.38µs 12.6 MElem/sec
fibonacci_loop/our/512 1.01 39.2±0.04µs 12.5 MElem/sec 1.00 38.8±0.05µs 12.6 MElem/sec
fibonacci_loop/our/524288 1.00 39.8±0.02ms 12.5 MElem/sec 1.00 39.8±0.04ms 12.6 MElem/sec
fibonacci_loop/our/64 1.01 5.0±0.01µs 12.1 MElem/sec 1.00 5.0±0.01µs 12.3 MElem/sec
fibonacci_loop/our/65536 1.03 5.1±0.00ms 12.3 MElem/sec 1.00 4.9±0.00ms 12.6 MElem/sec
fibonacci_loop/our/8 1.02 742.2±1.06ns 10.3 MElem/sec 1.00 727.0±0.90ns 10.5 MElem/sec
fibonacci_loop/our/8192 1.01 623.9±1.02µs 12.5 MElem/sec 1.00 619.0±0.59µs 12.6 MElem/sec
fibonacci_recursive/our/1 1.00 222.6±1.16ns 4.3 MElem/sec 1.15 255.8±13.62ns 3.7 MElem/sec
fibonacci_recursive/our/128 1.00 11.3±0.02µs 10.8 MElem/sec 1.21 13.6±0.26µs 8.9 MElem/sec
fibonacci_recursive/our/16 1.00 1735.4±3.49ns 8.8 MElem/sec 1.11 1932.2±45.86ns 7.9 MElem/sec
fibonacci_recursive/our/2 1.00 342.7±2.06ns 5.6 MElem/sec 1.09 371.9±13.99ns 5.1 MElem/sec
fibonacci_recursive/our/256 1.00 22.2±0.04µs 11.0 MElem/sec 1.35 29.8±0.97µs 8.2 MElem/sec
fibonacci_recursive/our/32 1.00 3.1±0.01µs 9.7 MElem/sec 1.32 4.1±0.06µs 7.4 MElem/sec
fibonacci_recursive/our/4 1.00 561.4±4.24ns 6.8 MElem/sec 1.09 610.3±24.04ns 6.3 MElem/sec
fibonacci_recursive/our/512 1.00 43.7±0.06µs 11.2 MElem/sec 1.32 57.8±0.85µs 8.4 MElem/sec
fibonacci_recursive/our/64 1.00 5.9±0.01µs 10.4 MElem/sec 1.31 7.7±0.56µs 8.0 MElem/sec
fibonacci_recursive/our/8 1.00 951.8±1.61ns 8.0 MElem/sec 1.12 1070.3±23.31ns 7.1 MElem/sec
```
Unfortunately, the order of the groups is lexically sorted (I believe?), but it would be way nicer for it to be human-readable numerically sorted:
```console
group benchmark-current.baseline benchmark-main.baseline
----- -------------------------- -----------------------
fibonacci_loop/our/1 1.06 196.4±0.37ns 4.9 MElem/sec 1.00 185.8±4.38ns 5.1 MElem/sec
fibonacci_loop/our/2 1.04 282.9±1.63ns 6.7 MElem/sec 1.00 271.3±2.37ns 7.0 MElem/sec
fibonacci_loop/our/4 1.01 433.6±0.98ns 8.8 MElem/sec 1.00 428.4±0.66ns 8.9 MElem/sec
fibonacci_loop/our/8 1.02 742.2±1.06ns 10.3 MElem/sec 1.00 727.0±0.90ns 10.5 MElem/sec
fibonacci_loop/our/16 1.00 1346.4±1.96ns 11.3 MElem/sec 1.00 1352.6±1.23ns 11.3 MElem/sec
fibonacci_loop/our/32 1.01 2.6±0.00µs 11.9 MElem/sec 1.00 2.5±0.00µs 12.0 MElem/sec
fibonacci_loop/our/64 1.01 5.0±0.01µs 12.1 MElem/sec 1.00 5.0±0.01µs 12.3 MElem/sec
fibonacci_loop/our/128 1.00 9.9±0.05µs 12.4 MElem/sec 1.00 9.8±0.01µs 12.4 MElem/sec
fibonacci_loop/our/256 1.01 19.6±0.02µs 12.4 MElem/sec 1.00 19.5±0.03µs 12.5 MElem/sec
fibonacci_loop/our/512 1.01 39.2±0.04µs 12.5 MElem/sec 1.00 38.8±0.05µs 12.6 MElem/sec
fibonacci_loop/our/1024 1.01 78.3±0.07µs 12.5 MElem/sec 1.00 77.5±0.09µs 12.6 MElem/sec
fibonacci_loop/our/2048 1.02 158.5±0.15µs 12.3 MElem/sec 1.00 154.8±0.14µs 12.6 MElem/sec
fibonacci_loop/our/4096 1.04 321.8±36.30µs 12.1 MElem/sec 1.00 309.5±0.38µs 12.6 MElem/sec
fibonacci_loop/our/8192 1.01 623.9±1.02µs 12.5 MElem/sec 1.00 619.0±0.59µs 12.6 MElem/sec
fibonacci_loop/our/16384 1.01 1246.7±1.13µs 12.5 MElem/sec 1.00 1237.5±1.13µs 12.6 MElem/sec
fibonacci_loop/our/32768 1.01 2.5±0.00ms 12.5 MElem/sec 1.00 2.5±0.00ms 12.6 MElem/sec
fibonacci_loop/our/65536 1.03 5.1±0.00ms 12.3 MElem/sec 1.00 4.9±0.00ms 12.6 MElem/sec
fibonacci_loop/our/131072 1.00 10.0±0.01ms 12.6 MElem/sec 1.01 10.1±0.01ms 12.4 MElem/sec
fibonacci_loop/our/262144 1.00 19.9±0.04ms 12.5 MElem/sec 1.01 20.1±0.02ms 12.5 MElem/sec
fibonacci_loop/our/524288 1.00 39.8±0.02ms 12.5 MElem/sec 1.00 39.8±0.04ms 12.6 MElem/sec
fibonacci_loop/our/1048576 1.01 80.5±0.27ms 12.4 MElem/sec 1.00 79.9±0.21ms 12.5 MElem/sec
fibonacci_recursive/our/1 1.00 222.6±1.16ns 4.3 MElem/sec 1.15 255.8±13.62ns 3.7 MElem/sec
fibonacci_recursive/our/2 1.00 342.7±2.06ns 5.6 MElem/sec 1.09 371.9±13.99ns 5.1 MElem/sec
fibonacci_recursive/our/4 1.00 561.4±4.24ns 6.8 MElem/sec 1.09 610.3±24.04ns 6.3 MElem/sec
fibonacci_recursive/our/8 1.00 951.8±1.61ns 8.0 MElem/sec 1.12 1070.3±23.31ns 7.1 MElem/sec
fibonacci_recursive/our/16 1.00 1735.4±3.49ns 8.8 MElem/sec 1.11 1932.2±45.86ns 7.9 MElem/sec
fibonacci_recursive/our/32 1.00 3.1±0.01µs 9.7 MElem/sec 1.32 4.1±0.06µs 7.4 MElem/sec
fibonacci_recursive/our/64 1.00 5.9±0.01µs 10.4 MElem/sec 1.31 7.7±0.56µs 8.0 MElem/sec
fibonacci_recursive/our/128 1.00 11.3±0.02µs 10.8 MElem/sec 1.21 13.6±0.26µs 8.9 MElem/sec
fibonacci_recursive/our/256 1.00 22.2±0.04µs 11.0 MElem/sec 1.35 29.8±0.97µs 8.2 MElem/sec
fibonacci_recursive/our/512 1.00 43.7±0.06µs 11.2 MElem/sec 1.32 57.8±0.85µs 8.4 MElem/sec
```
(this output was created using `sort --stable --field-separator=/ --key=1,2d --key=3g` plus manual fixing of the first two lines).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how critcmp orders benchmark groups before rendering this output, then run it against the benchmark result shown in the issue. The work is done when numeric components such as 1, 2, 4, 8, and 16 appear in human-readable order within each group, while the existing output remains correctly formatted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, performance
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100