google / google/fuzzbench

Cross benchmark pairwise differential coverage plot

Open
#741 0 comments 1 reaction 0 assignees View on GitHub
minor enhancement
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

It would be useful too have a visualization showing which fuzzers complement each other well in general, across benchmarks.

Not clear, but an aggregate / cross-benchmark version of the current per-benchmark pairwise unique coverage heatmap plots *might* help with this. While the current per-benchmark plots show the number of edges covered by fuzzer 1 but not by fuzzer 2, we could compute the aggregate matrix plot as follows:

- for each benchmark, and for each fuzzer pair (f1, f2), compute the ratio of the unique and combined coverage,
ie. ratio_{bench,f1,f2} = C(f1)\C(f2) / C(f1)UC(f2).
- then for the aggregate matrix, for each fuzzer pair (f1, f2), take the average of the per-benchmark ratios
ie. agg_ratio_{f1,f2} = AVG_{bench in all benchmarks}(ratio_{benc,f1,f2})

Prior discussion: https://github.com/google/fuzzbench/issues/719#issuecomment-692940683

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.