When is criterion stable between runs, how can it be made more so?
- Dominant language
- Haskell
- Stars
- 523
- Forks
- 89
- PR merge metrics
- No merged PRs in 30d
Description
Criterion gives highly precise measurements. Given two measurements of a _simple_ microbenchmark `A`, `A1`,`A2`, if:
- both were taking starting from a similar machine state,
- both report very high R^2 values, and
- both were run for a long `-L20` or higher,
then A1/A2 should be close estimates, right? No, unfortunately.
When measuring 1279 benchmarks on Stackage, we have found that it's very common to have greater than 10% variation between consecutive runs of the same, small, deterministic benchmark.
Anecdotally, we seem to get more stable numbers from individual high `--iters` runs, than from linear regression. I don't have a good explanation yet. Perhaps the non-determinism in the selection of data points (on the X axis) is having more of an effect than we expected? Certainly, when there is a bad R^2, we've seen exactly where it starts running has a big effect.
@RyanGlScott and @vollmerm have been working on this.
(On a related note, it would be great to have some assistance when using criterion with the [kinds of things Krun controls](https://github.com/softdevteam/krun), like waiting for the machine to cool down to a baseline temperature before starting a run.)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.