More than one benchmark category?
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
It would be nice to have different benchmark categories
1. to accomodate specialized fuzzers (structure-aware fuzzers, or fuzzers that find special kindss of bugs), and
2. to address the overfitting problem where fuzzers are optimized to a specific benchmark set (maybe at the cost of general efficiency).
A *fuzzer* could maintain a deny or allow list for certain benchmark categories (e.g., [structure-aware fuzzers](https://github.com/google/fuzzbench/blob/master/fuzzers/aflsmart/README.md) might only opt in for subjects where a grammar orprotobuf is available). An *evaluation* could specify which benchmark categories to run. One *benchmark* could be member in more than one category (but would still be run only once if selected in multiple categories).
For instance, apart from specialized categories for specialized fuzzers, I would love to see a category that randomly chooses 10 or more OSS-Fuzz projects every time it is run, preventing overfitting and giving a more realistic picture of a fuzzers overall performance on a random open-source program. The current, deliberately chosen benchmark set could remain as one category, of course. Individual bug benchmarks (even specific classes of bugs) could get their own category.
//cc @thuanpv
Contributor guide
Assessment
This issue has not been assessed yet.