dotnet / dotnet/BenchmarkDotNet
ELIA5 - How to translate the result into (old school?) wallclock time comparison of average runs
- Dominant language
- C#
- Stars
- 11.5k
- Forks
- 1.1k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 12
Description
Some results produced by BenchmarkDotNet have header like: `Mean | Error | StdDev | Scaled ..`. I have seen people explaining to others in GitHub prs and issues what the output means. I am tired of pretending that I understand this output. Wikipedia is a no-go, read this stuff few times to jog my statisticy muscles but fail to make relationship with what i read with what someone has posted, a fancy table in PR description.
There is traditional way of benchmarking, where the wallclock execution time is captured for each run and the average difference is shown in the result. That is very clear to me and it does not take any head-scratching and re-reading the results multiple times to make sense out of those numbers.
Since BenchmarkDotNet is becoming a defacto standard (various results posted in the wild: blogs, github issues & PR), please add a section in README or Documentation for dummies explaining the basics of reading the result (preferably with example taken from one of the PR from https://github.com/dotnet/coreclr/pulls). In v.next, a small `(?)` at the bottom or top corner of result table linking to that page would be great.
RE: https://github.com/dotnet/BenchmarkDotNet/issues/349
this time, please don't forget about "Error" and "Scaled" :)
Contributor guide
Assessment
This issue has not been assessed yet.