Number of benchmarks in critical difference plot
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
I tried to understand how plotting critical difference in fuzzbench works and I see that you use [number of benchmarks](https://github.com/google/fuzzbench/blob/master/analysis/plotting.py#L479) when generating a plot. From what I understood, if N is meant to be the number of datasets then I think this should instead be the _number of trials_ in our case?
For example given two experiments (trials=10 and trials=100) having the same ranking, should the experiment with 100 trials have smaller critical difference?
Trials = 10

Trials = 100

Contributor guide
Assessment
This issue has not been assessed yet.