google / google/fuzzbench

Effects of the collision problem in AFL (and variants thereof) on the coverage evaluations

Open
#1,250 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
302
PR merge metrics
No merged PRs in 30d

Description

Hi everybody,

I wonder to what extent the _branch collision problem_ in AFL (and most of its variants) affects the code coverage evaluations of FuzzBench. If I understand correctly, the branch collision problem causes AFL to reject coverage-increasing inputs, i.e. they are not added to the input queue. This also means that FuzzBench cannot capture these inputs, which can lead to inaccurate coverage results and thus biased evaluations, especially when comparing AFL-based fuzzers to non-AFL-based fuzzers.

In the paper "CollAFL: Path Sensitive Fuzzing", the authors say that for certain programs, up to **75%** of the edges could collide with others, which in my eyes would lead to a large bias in your evaluations.

Do you have any numbers on how big the deviations are and thus whether this is negligible?

Best regards,
Stephan

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.