benchmark-action / benchmark-action/github-action-benchmark

Feature request: Support iai Rust benchmarks

Open
#108 3 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
1.3k
Forks
184
PR merge metrics
No merged PRs in 30d

Description

Rust benchmarking can be noisy in virtualized environments (GH Actions) and the best way to run the benchmarks in those is probably [iai](https://github.com/bheisler/iai). Its format seems to be unsupported my workflow runs the benchmarks and prints them out to file : https://github.com/kirillbobyrev/pabi/runs/5079239022

From the [docs](https://bheisler.github.io/criterion.rs/book/iai/iai.html):

> It is intended as a complement to Criterion.rs; among other things, it's useful for reliable benchmarking in CI.

And

> For benchmarks that run in CI (especially if you're checking for performance regressions in pull requests on cloud CI) you should use Iai. For benchmarking on Windows or other platforms that Valgrind doesn't support, you should use Criterion-rs. For other cases, I would advise using both. Iai gives more precision and scales better to larger benchmarks, while Criterion-rs allows for excluding setup time and gives you more information about the actual time your code takes and how strongly that is affected by non-determinism like threading or hash-table randomization. If you absolutely need to pick one or the other though, Iai is probably the one to go with.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.