google / google/fuzzbench

Speed up measuring

Open
#164 0 comments 0 reactions 0 assignees View on GitHub
enhancement priority
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

Measuring is currently a bottleneck that slows done the entire experiment cycle. Experiments are taking a day or two to "complete" even after the last trial terminates because measuring is slow. There are a couple ways measuring can be improved to fix this situation:

1. Currently measuring happens on one (very large) machine. If it ran on multiple machines it could be scaled horizontally.

2. Currently measuring works by:
i. Syncing the entire data directory
ii. Processing the corpora in parallel
iii. Go to i.

This means that if some benchmarks are slower to process than others, cores will be idle while corpora for those benchmarks are being measured.

I've had luck with copying each corpus archive in parallel and then processing it. This might break things since gsutil frequently has issues when used in parallel, but I think it's worth trying.

3. Currently we only measure test cases in a corpus archive that wasn't in the previous archive. This means that corpus archives in a trial must be measured sequentially. We can maybe improve things by letting archives be measured in parallel.

4. Eliminating slow benchmarks. Although I haven't analyzed this thoroughly, it seems like certain benchmarks slow down the entire measuring process by being much slower than everything else. This can pose a problem for accuracy as well since we won't keep measuring a corpus that takes longer than 5 minutes.

5. Encourage better use of output_corpus: Currently AFL based fuzzers put their entire output directory in output_corpus, meaning that the measurers need to process hangs and crashes (which we will want to support ).

Solving 2-3 will help with 1 which frankly should be done anyway. I certainly don't plan on implementing all of these solutions, just enough that this problem is solved.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.