haskell / haskell/criterion

When is criterion stable between runs, how can it be made more so?

Open
#115 1 comment 0 reactions 0 assignees View on GitHub
Question
Dominant language
Haskell
Stars
523
Forks
89
PR merge metrics
No merged PRs in 30d

Description

Criterion gives highly precise measurements. Given two measurements of a _simple_ microbenchmark `A`, `A1`,`A2`, if:
- both were taking starting from a similar machine state,
- both report very high R^2 values, and
- both were run for a long `-L20` or higher,

then A1/A2 should be close estimates, right? No, unfortunately.

When measuring 1279 benchmarks on Stackage, we have found that it's very common to have greater than 10% variation between consecutive runs of the same, small, deterministic benchmark.

Anecdotally, we seem to get more stable numbers from individual high `--iters` runs, than from linear regression. I don't have a good explanation yet. Perhaps the non-determinism in the selection of data points (on the X axis) is having more of an effect than we expected? Certainly, when there is a bad R^2, we've seen exactly where it starts running has a big effect.

@RyanGlScott and @vollmerm have been working on this.

(On a related note, it would be great to have some assistance when using criterion with the [kinds of things Krun controls](https://github.com/softdevteam/krun), like waiting for the machine to cool down to a baseline temperature before starting a run.)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.