google / google/fuzzbench

Number of benchmarks in critical difference plot

Open
#1,417 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

I tried to understand how plotting critical difference in fuzzbench works and I see that you use [number of benchmarks](https://github.com/google/fuzzbench/blob/master/analysis/plotting.py#L479) when generating a plot. From what I understood, if N is meant to be the number of datasets then I think this should instead be the _number of trials_ in our case?

For example given two experiments (trials=10 and trials=100) having the same ranking, should the experiment with 100 trials have smaller critical difference?

Trials = 10
![10](https://user-images.githubusercontent.com/16756648/176852171-fe593bf2-71b2-43b3-bc3d-e9c014ccbb8f.png)

Trials = 100
![100](https://user-images.githubusercontent.com/16756648/176852250-16968376-fcaf-4af8-970e-d5162e00eaae.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.