google / google/fuzzbench

More detailed output statistics.

Open
#1,253 16 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

As an end user, it’s the number of defects found which is of interest and hence the number of paths discovered is the best proxy for this and duly show in the fuzzbench reports.

As a developer, there are two main ways to improve a fuzzer. Either increase its performance (iterations per second) or increase the yield (paths found per iteration, fractional) by introducing new techniques. As a developer, it would be really useful if these statistics could also be graphed in the fuzzbench output.

This could allow you to answer the following questions:
Have my changes improved the speed of the fuzzer?
Is the speed of the fuzzer consistent throughout the 23h run? Is it consistent between runs?
Is the relative speed of my fuzzer better or worse for some benchmarks compared to another fuzzer?
How much has my new fancy technique improved the yield? How bad has the associated performance overhead been?

Obviously these stats are meaningless on their own, having the fastest fuzzer means nothing if the yield is poor and viceversa. But I think these stats could provide very useful insights for developers.

Does this sound possible? What are your thoughts?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.