google / google/fuzzbench

Support tracks to compare classes of fuzzers (binary-only, source-only, hybrids, etc)

Open
#76 8 comments 2 reactions 0 assignees View on GitHub
core feature
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

Thanks for maintaing an awesome project!

I am writing to discuss my concern about the unfairness of fuzzbench results due to the difference between binary-level vs. source-level fuzzers/

When I look at the current [sample report](https://www.fuzzbench.com/reports/sample/index.html), all the tools used here except Eclipser run with source-level instrumentation (with afl-cc). Eclipser, on the other hand, uses QEMU to instrument binaries.

It is well-known that binary-level instrumentation incurs significant overhead (several orders of magnitude) compared to source-level instrumentation. Therefore, comparing Eclipser with source-level fuzzers, e.g., AFL, is not entirely fair as they have different goals and uses. However, comparing Eclipser with AFL running in the QEMU mode (-Q option) would be fair, for example.

So I would like to suggest separating tracks in fuzzbench into two: binary track, and source track. In the binary track, we can include AFL-QEMU, Eclipser, VUzzer, etc. I believe showing two sets of graphs for each program would be enough. For your information, having multiple tracks in comparing tools is a common practice in other domains. For example, SMT-COMP currently has 6 tracks: https://smt-comp.github.io/2019/results.html.

This way, people can appreciate more about binary-level fuzzing research 😄 I truly believe this will benefit our community as well.

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.