Coverage measurements invalidated by non-testcase fuzzer outputs
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
Fuzzbench's approach to coverage measurement simply runs the coverage of every file in the fuzzer corpus directory. However, most fuzzers (e.g. all of the AFL++ versions) often simply place their `output directory` into this directory. This fundamentally skews the coverage for those fuzzers on benchmark that parse related file formats.
A simple case of this is seen in [my fuzzbench experiment](https://www.fuzzbench.com/reports/experimental/2023-04-27-symcts/index.html). The large boost my tools achieve at the beginning for `bloaty_fuzz_target` is because I include a copy of all of the compiled binaries in a hidden `.run_info` directory to be able to reproduce the experiment locally.
The instrumented files are ELF files, and `bloaty_fuzz_target` parses ELF files. Another more sneaky issue arises similarly **if a fuzzer crashes and dumps a corefile into this directory** so the maintainers can reproduce why it crashed (as for example `symsan` and `aflplusplus` do), **this corefile will itself be measured towards the fuzzer's coverage** (because it is a valid ELF file), and a crashing fuzzer can significantly benefit from it, at least in the case of `bloaty`. I have observed this at least once for `SymSan`. However, other scenarios are possible, for example a fuzzer dumping some status as JSON files while fuzzing `jsoncpp`, etc.
At the moment however, it is simply not possible to get information out of the experiment without having it be treated as a testcase (other than dumping it to stdout, which is a bad idea). As a solution I propose to either
1. Allow for the exclusion of hidden files (prefixed by a `.`), and modify the fuzzers to run inside a `.cwd` directory or something of that sort
2. Add a way for fuzzers to specify `glob`s or `path`s which contain real testcases to be measured, and only measure those
3. Split out a separate file collection mechanisms only for testcases, so that maintainers can retrieve debug information and logging without having them be interpreted as testcases
[I implemented 1. in my fork](https://github.com/Lukas-Dresel/fuzzbench/commit/926dbb76f1741c018664688de5a5c81ac148808d) because it is simple to do, and none of the fuzzers I investigated stored crashes or testcases in a hidden directory. This also helps improve the performance of the measurements since unrelated `.redundant_edges` and `.auto_extras` in AFL++ for example aren't included (and shouldn't have been anyways)
Contributor guide
Assessment
This issue has not been assessed yet.