google / google/fuzzbench

More than one benchmark category?

Open
#727 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

It would be nice to have different benchmark categories
1. to accomodate specialized fuzzers (structure-aware fuzzers, or fuzzers that find special kindss of bugs), and
2. to address the overfitting problem where fuzzers are optimized to a specific benchmark set (maybe at the cost of general efficiency).

A *fuzzer* could maintain a deny or allow list for certain benchmark categories (e.g., [structure-aware fuzzers](https://github.com/google/fuzzbench/blob/master/fuzzers/aflsmart/README.md) might only opt in for subjects where a grammar orprotobuf is available). An *evaluation* could specify which benchmark categories to run. One *benchmark* could be member in more than one category (but would still be run only once if selected in multiple categories).

For instance, apart from specialized categories for specialized fuzzers, I would love to see a category that randomly chooses 10 or more OSS-Fuzz projects every time it is run, preventing overfitting and giving a more realistic picture of a fuzzers overall performance on a random open-source program. The current, deliberately chosen benchmark set could remain as one category, of course. Individual bug benchmarks (even specific classes of bugs) could get their own category.

//cc @thuanpv

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.