Roadmap for future features
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
To keep track of the desirable features that we'd like to add in the future, here is a list of them in the order of priority:
1. [x] [Make CI tests OSS-Fuzz-like trial builds](https://github.com/google/fuzzbench/issues/1556).
2. [x] Revive more bug-based benchmarks and test them thoroughly with core fuzzers.
3. [ ] Save experiment logs to an externally accessible bucket for fuzzer owners to debug.
5. [ ] Versionize FuzzBench: We use a stable version of FuzzBench for production (experiments, etc.) and periodically replace it with new versions.
6. [x] Update benchmarks yearly to ensure the timeliness of FuzzBench.
7. [ ] Allow users to add fuzzer flags from the command line or configuration file.
8. [x] [Parallelize the coverage evaluation process](https://github.com/google/fuzzbench/issues/1506) to save time, fixed by #1672.
9. [ ] [More sophisticated ranking for bug-based benchmarks](https://github.com/google/fuzzbench/pull/1765).
10. [ ] When a fuzzer instance crashes during an experiment, fill the rest of the experiment with its latest data before the crash. This avoids sudden increases/drops in performance in the cov/bug vs. time plot (and avoids cheating).
11. [ ] A script to run private experiment. Given a patch file that contains the new fuzzer, the script applies the patch, requests a private cloud experiment, and print the link to the experiment data and result.
12. [ ] Notify us when an experiment finishes or crashes. This can be done via email.
## Maintenance
1. [x] Omit empty strings in /gcbrun commands.
* For example, [this command](https://pantheon.corp.google.com/storage/browser/fuzzbench-data/2023-04-08-aflpp-um?project=fuzzbench&pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&forceOnObjectsSortingFiltering=false) failed due to a mysterious extra space between `--fuzzers` and the first fuzzer. After being converted to a list, the command becomes `[..., '--fuzzers', '', 'aflplusplus_um_parallel']`, making our script think that we passed an empty string as a fuzzer's name.
* The solution is to remove all empty strings in the list.
Contributor guide
Assessment
This issue has not been assessed yet.