google / google/atheris

Hypothesis produces poor results

Open
#20 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
1.7k
Forks
127
PR merge metrics
No merged PRs in 30d

Description

I'm in the process of writing end-to-end tests to make sure Python coverage is high-quality. In doing so, I discovered that Hypothesis structured fuzzing causes really poor fuzz quality - even the example in the readme doesn't work:

```python
import atheris
from hypothesis import given, strategies as st

@given(st.from_regex(r"\w+!?", fullmatch=True))
@atheris.instrument_func
def test(string):
assert string != "bad"

atheris.Setup(sys.argv, atheris.instrument_func(test.hypothesis.fuzz_one_input))
atheris.Fuzz()
```

I checked, and this isn't caused by the new coverage method - this works poorly with old coverage too. Doing this with regular Atheris, however, works excellently.

@Zac-HD, as the original contributor of the Hypothesis examples: do you have any suggestions here? I was thinking something along the lines of an external mutator for libFuzzer might work to fix the issues here. That's how libprotobuf-mutator for C++ works.
@nedwill your input might also be helpful here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.