A new model for requesting experiments
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
We should change the model we use to request experiments.
Instead of committing all code to be used in experiments along with a request in experiment-requests.yaml, we will start experiments on pull requests using the gcbrun keyword.
This will have the following benefits:
* Experiments will start immediately instead of at 6 AM and 6 PM PST
* Experiments will be trivial to reproduce, just a comment (unless there's a db schema change...or some other breaking change this will need to be backported)
* It will make fuzzbench more flexible. Experiments can go wild making changes to the fuzzbench infra. Now experiments will be able to add whatever benchmarks they want (and not worry if every other fuzzer in the world can support it. They can now do other interesting things like change the snapshot time from 15 minutes to arbitrary times. Change the OS if you want. I'm actually excited about what people will come up here, I feel like FuzzBench hasn't accepted so many innovations in these areas because we are worried about the effects this will have on others. Now we can provide a solid framework and CPU time and let others do the thinking.
* It will be more obvious when experiments fail to start, the current model uses a single instance with a cron job and is quite fragile.
* It will reduce maintenance burden on fuzzbench maintainers. I think in hindsight it was a mistake to commit every fuzzer to fuzzbench and give some kind of guarantee that we will maintain them. In the future we should only commit core fuzzers, variants will be kept in people's forks and to experiment on them they can make a PR. This will also make our CI situation a little less ridiculous and wasteful.
I can think of some downsides however:
* More fragmentation. If researcher A does an interesting experiment with fuzzer B and then researcher C wants to use fuzzer A in their experiment, they now have to grab that integration code from a pull request instead of the master branch. I don't think this use case is super common however.
* Possibly slightly increased burden on power users that will need to do many different kinds of experiments. Now they may need to deal with merging our code.
Contributor guide
Assessment
This issue has not been assessed yet.