google / google/fuzzbench

Coverage measurements invalidated by non-testcase fuzzer outputs

Open
#1,836 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

Fuzzbench's approach to coverage measurement simply runs the coverage of every file in the fuzzer corpus directory. However, most fuzzers (e.g. all of the AFL++ versions) often simply place their `output directory` into this directory. This fundamentally skews the coverage for those fuzzers on benchmark that parse related file formats.

A simple case of this is seen in [my fuzzbench experiment](https://www.fuzzbench.com/reports/experimental/2023-04-27-symcts/index.html). The large boost my tools achieve at the beginning for `bloaty_fuzz_target` is because I include a copy of all of the compiled binaries in a hidden `.run_info` directory to be able to reproduce the experiment locally.

The instrumented files are ELF files, and `bloaty_fuzz_target` parses ELF files. Another more sneaky issue arises similarly **if a fuzzer crashes and dumps a corefile into this directory** so the maintainers can reproduce why it crashed (as for example `symsan` and `aflplusplus` do), **this corefile will itself be measured towards the fuzzer's coverage** (because it is a valid ELF file), and a crashing fuzzer can significantly benefit from it, at least in the case of `bloaty`. I have observed this at least once for `SymSan`. However, other scenarios are possible, for example a fuzzer dumping some status as JSON files while fuzzing `jsoncpp`, etc.

At the moment however, it is simply not possible to get information out of the experiment without having it be treated as a testcase (other than dumping it to stdout, which is a bad idea). As a solution I propose to either

1. Allow for the exclusion of hidden files (prefixed by a `.`), and modify the fuzzers to run inside a `.cwd` directory or something of that sort
2. Add a way for fuzzers to specify `glob`s or `path`s which contain real testcases to be measured, and only measure those
3. Split out a separate file collection mechanisms only for testcases, so that maintainers can retrieve debug information and logging without having them be interpreted as testcases

[I implemented 1. in my fork](https://github.com/Lukas-Dresel/fuzzbench/commit/926dbb76f1741c018664688de5a5c81ac148808d) because it is simple to do, and none of the fuzzers I investigated stored crashes or testcases in a hidden directory. This also helps improve the performance of the measurements since unrelated `.redundant_edges` and `.auto_extras` in AFL++ for example aren't included (and shouldn't have been anyways)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.