google / google/fuzzbench

Measure time to achieve a given coverage (not vice versa)

Open
#327 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

For each new branch to be covered (or anything new to be [dis]covered, really), it takes exponentially more time (write up with experiments to be published shortly). This is why a comparison of the *coverage achieved by each fuzzer within a fixed time budget* gives very small deltas. This relationship might result in potentially spurious differences when ranking fuzzers in terms of their efficiency.

Instead, I would suggest to measure *the time each fuzzer takes to achieve a given coverage*. The *given* coverage could be determined manually or using the coverage achieved in 24h by that fuzzer which achieved the smallest coverage in 24h. This should give larger differences across fuzzers and might facilitate a better assessment of relative fuzzer efficiency.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.