pylint-dev / pylint-dev/astroid
Add property-based tests and fuzzing
- Dominant language
- Python
- Stars
- 582
- Forks
- 357
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 23
Description
I'd like to explore adding property-based tests with fuzzing tools like http://jwilk.net/software/python-afl, https://hypothesis.readthedocs.org/en/latest/, and https://bitbucket.org/ebo/pyfuzz. Here are some possible tests:
- Feed random Python code to the rebuilder to find crash bugs.
- Test to_string() and the rebuilder by running AST->code->AST transformations with the invariant that the AST must be the same in both cases.
- Find any more cases where there are non-AST objects in the ASTs with induction.
- Establish that the inference is idempotent.
- Check the inference doesn't crash.
- Verify that traversing trees is reliable, e.g. that going up (calling parent) and then down returns the same node.
I don't know how well this will work because language fuzzing is hard in general, but I think it's worth trying. For astroid in particular, generating random ASTs, which is easier than trying to generate Python code though far from easy, might work. We may want to consider, in addition to trying to generate random data, running some of these same properties on a large corpus of Python source code files. One obvious candidate is the standard library, but there's a lot of Python code out there we could potentially use. David MacIver used a corpus-based approach to test https://github.com/google/yapf/ with some efficacy: http://www.drmaciver.com/2015/03/27-bugs-in-24-hours/.
In general, I'd like to try push astroid testing toward defining what the code should do and then looking for falsifying examples in some automated fashion. I think we could also simplify some of the existing tests using this approach.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the rebuilder, to_string(), inference, and parent traversal named in the issue. Decide whether generated ASTs or a standard-library corpus is the initial input, then define a reproducible property test and document what a passing invariant or crash check demonstrates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100