dotnet / dotnet/BenchmarkDotNet

An easy way to check for statistically significant difference between benchmarks

Open
#786 2 comments 4 reactions 1 assignee Claimed by @AndreyAkinshin View on GitHub
Area:Engine
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
4d 11h
Merged PRs (30d)
11

Description

Currently, we have optional `WelchTTestPValueColumn` which help you to verify that there is a statistically significant difference between benchmarks. However, it doesn't work great with default run strategy because this strategy typically doesn't perform enough iterations. Users have to manually choose a satisfactory amount of iterations. Thus, it's possible to do such checks, but the user experience is not good enough. We can do the following:

* Introduce additional property in `AccuracyMode`. Let's call it `StopСriterion` (let me know if you have better ideas about naming). It will contain logic which should decide when do we have enough iterations.
* Currently, we have hardcoded logic inside `EngineTargetStage`. Let's move it to a class called `StdErrStopCriterion`.
* We can introduce `WelchStopCriterion` which will do additional iterations until we sure that it's enough for the Welch's Two Sample t-test. (Bonus: users will be able to write own criterion)
* `StopCriterion` should be able to affect `IOrderProvider.GetExecutionOrder` and ask to run baseline benchmarks first.
* `EngineTargetStage.RunAuto` should get additional information about benchmarks like `IsBaseline` value. The non-baseline benchmarks should get all measurements from the baseline benchmark in the corresponded group.

Original request: https://twitter.com/AnthonyLloyd123/status/1005388154046644230

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.