google / google/fuzzbench

Roadmap for future features

Open
#1,590 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

To keep track of the desirable features that we'd like to add in the future, here is a list of them in the order of priority:
1. [x] [Make CI tests OSS-Fuzz-like trial builds](https://github.com/google/fuzzbench/issues/1556).
2. [x] Revive more bug-based benchmarks and test them thoroughly with core fuzzers.
3. [ ] Save experiment logs to an externally accessible bucket for fuzzer owners to debug.
5. [ ] Versionize FuzzBench: We use a stable version of FuzzBench for production (experiments, etc.) and periodically replace it with new versions.
6. [x] Update benchmarks yearly to ensure the timeliness of FuzzBench.
7. [ ] Allow users to add fuzzer flags from the command line or configuration file.
8. [x] [Parallelize the coverage evaluation process](https://github.com/google/fuzzbench/issues/1506) to save time, fixed by #1672.
9. [ ] [More sophisticated ranking for bug-based benchmarks](https://github.com/google/fuzzbench/pull/1765).
10. [ ] When a fuzzer instance crashes during an experiment, fill the rest of the experiment with its latest data before the crash. This avoids sudden increases/drops in performance in the cov/bug vs. time plot (and avoids cheating).
11. [ ] A script to run private experiment. Given a patch file that contains the new fuzzer, the script applies the patch, requests a private cloud experiment, and print the link to the experiment data and result.
12. [ ] Notify us when an experiment finishes or crashes. This can be done via email.

## Maintenance
1. [x] Omit empty strings in /gcbrun commands.
* For example, [this command](https://pantheon.corp.google.com/storage/browser/fuzzbench-data/2023-04-08-aflpp-um?project=fuzzbench&pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false) failed due to a mysterious extra space between `--fuzzers` and the first fuzzer. After being converted to a list, the command becomes `[..., '--fuzzers', '', 'aflplusplus_um_parallel']`, making our script think that we passed an empty string as a fuzzer's name.
* The solution is to remove all empty strings in the list.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.