Support measuring/reporting for "unmanaged" trials.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
FuzzBench assumes that it can build and run trials for everything it needs to measure.
However, there are some fuzzers (@gamozolabs tko) that FuzzBench probably can't build and run.
I think it would be very interesting for fuzzbench to be able to measure performance of these fuzzers on our benchmarks.
E.g. someone who wants to benchmark these fuzzers can just give us the corpus archives and the measurer/report basically treats it like any fuzzer.
At least for now, all we would need to do to support it is some kind of way of telling fuzzbench the fuzzer is unmanaged (so create db entities for it but don't try building or running) and then the "unmanaged" trials can simply put their corpus archives on Google Cloud Storage so they can be measured and compared to our other fuzzers.
As Brandon points out, this would make cheating easy so some kind of asterisk and explanation is probably needed in the report.
I think even without intentional cheating this comparison is problematic because the CPU family the unmanaged trials run on is probably different from the ones we use for the "managed" (regular) ones. But I still think there is value here. Especially when we have differences like libpcap_fuzz_both, I'm sure something like double or triple the CPU time isn't going to make fuzzers like AFL do better than the other ones.
Contributor guide
Assessment
This issue has not been assessed yet.