Sampling Initial Seed Corpus and Analysis
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
TO @jonathanmetzman @lszekeres
CC @mboehme @inferno-chromium
We have two related features which we've implemented on a private fork that we'd like to integrate into Fuzzbench. The first is the ability to sample from a larger pool of seeds to provide a unique corpus to each fuzzer per trial during a benchmarking run. The second consists of additional data-analysis to give some insight into how various aspects of the initial corpora and programs under test might be affecting benchmarking outcomes.
The purpose of this issue it to establish the following:
1. [Sampling] We currently have a script we've been using for local experiments that samples from e.g. a project's OSS-Fuzz corpus to generate random initial corpora. We then mount those in the docker containers of the runners. We also kick off the first measurer cycle before launching the fuzzer process to grab the initial coverage of the corpus. Are there other considerations or another approach we should take for adding this feature?
2. [Properties] Which properties would you consider to be interesting? We currently have
* seed-corpus: initial coverage, number of seeds, average seed exec time, average seed size
* program: size (and others). Anything else that you would like to look at?
3. [UI/UX] What is the interface that you want to present to the users? For the seed sampling, probably additional field(s) in the YAML configuration file to select a sampling level / strategy? For the data presentation, we have produced several visualizations which show the relative impact of a particular property on the final ranking of a fuzzer or its coverage. We are happy to share these separately and would welcome any feedback you might have on where and how to present this data in a Fuzzbench report.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.