Variance in FuzzBench trials with same random seed
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
@vanhauser-thc wrote [here](https://github.com/google/fuzzbench/issues/654#issuecomment-672845457):
>the experiment has started => https://www.fuzzbench.com/reports/2020-08-12/index.html
and we can see the aflplusplus_same1 .. aflplusplus_same3 instances to have quite a different coverage.
>for several benchmarks this will even out as their overage saturate after 6 hours, so capturing this now is important.
>most easy to see for curl, re2 and sqlite3:
>|fuzzer|fuzzer1|fuzzer2|fuzzer3|
>|:--:|:--:|:--:|:--:|
>|curl | 17148.0 | 17161.5 | 17155.0|
>|re2| 3461.0 | 3455.0 | 3488.0|
>|sqlite3|26895.0 | 26611.5 | 26420.5|
>as all PRNG values are the same and input selection based on same values this shows that the instances are running with different speeds.
Contributor guide
Assessment
This issue has not been assessed yet.