dotnet / dotnet/BenchmarkDotNet

ELIA5 - How to translate the result into (old school?) wallclock time comparison of average runs

Open
#680 1 comment 0 reactions 1 assignee Claimed by @AndreyAkinshin View on GitHub
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
6d 11h
Merged PRs (30d)
12

Description

Some results produced by BenchmarkDotNet have header like: `Mean | Error | StdDev | Scaled ..`. I have seen people explaining to others in GitHub prs and issues what the output means. I am tired of pretending that I understand this output. Wikipedia is a no-go, read this stuff few times to jog my statisticy muscles but fail to make relationship with what i read with what someone has posted, a fancy table in PR description.

There is traditional way of benchmarking, where the wallclock execution time is captured for each run and the average difference is shown in the result. That is very clear to me and it does not take any head-scratching and re-reading the results multiple times to make sense out of those numbers.

Since BenchmarkDotNet is becoming a defacto standard (various results posted in the wild: blogs, github issues & PR), please add a section in README or Documentation for dummies explaining the basics of reading the result (preferably with example taken from one of the PR from https://github.com/dotnet/coreclr/pulls). In v.next, a small `(?)` at the bottom or top corner of result table linking to that page would be great.

RE: https://github.com/dotnet/BenchmarkDotNet/issues/349
this time, please don't forget about "Error" and "Scaled" :)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.