benchmark-action / benchmark-action/github-action-benchmark

Allow choice of benchmark stat used for `pytest`

Open
#92 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.3k
Forks
184
PR merge metrics
No merged PRs in 30d

Description

`pytest-benchmark` automatically computes a fairly comprehensive set of stats across multiple runs of each benchmark, including minimum, mean, median, etc. runtime, and Operations Per Second (calculated as the inverse of the mean runtime).

It's commonly recommended that benchmark comparisons use the minimum runtime, as it is more stable than other statistics. However, the actions are currently hardcoded to use the `ops` statistic.

Is it possible to make this configurable?

(This request may apply more widely than `pytest`, but I'm not familiar with the other output formats that are handled)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.