google / google/fuzzbench

Variance in FuzzBench trials with same random seed

Open
#667 31 comments 1 reaction 0 assignees View on GitHub
needs analysis
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

@vanhauser-thc wrote [here](https://github.com/google/fuzzbench/issues/654#issuecomment-672845457):
>the experiment has started => https://www.fuzzbench.com/reports/2020-08-12/index.html
and we can see the aflplusplus_same1 .. aflplusplus_same3 instances to have quite a different coverage.

>for several benchmarks this will even out as their overage saturate after 6 hours, so capturing this now is important.

>most easy to see for curl, re2 and sqlite3:

>|fuzzer|fuzzer1|fuzzer2|fuzzer3|
>|:--:|:--:|:--:|:--:|
>|curl | 17148.0 | 17161.5 | 17155.0|
>|re2| 3461.0 | 3455.0 | 3488.0|
>|sqlite3|26895.0 | 26611.5 | 26420.5|

>as all PRNG values are the same and input selection based on same values this shows that the instances are running with different speeds.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.