pylint-dev / pylint-dev/astroid

Add property-based tests and fuzzing

Open
#282 3 comments 0 reactions 0 assignees View on GitHub
Enhancement ✨
Dominant language
Python
Stars
582
Forks
357
Avg merge
1d 1h
Merged PRs (30d)
23

Description

I'd like to explore adding property-based tests with fuzzing tools like http://jwilk.net/software/python-afl, https://hypothesis.readthedocs.org/en/latest/, and https://bitbucket.org/ebo/pyfuzz. Here are some possible tests:
- Feed random Python code to the rebuilder to find crash bugs.
- Test to_string() and the rebuilder by running AST->code->AST transformations with the invariant that the AST must be the same in both cases.
- Find any more cases where there are non-AST objects in the ASTs with induction.
- Establish that the inference is idempotent.
- Check the inference doesn't crash.
- Verify that traversing trees is reliable, e.g. that going up (calling parent) and then down returns the same node.

I don't know how well this will work because language fuzzing is hard in general, but I think it's worth trying. For astroid in particular, generating random ASTs, which is easier than trying to generate Python code though far from easy, might work. We may want to consider, in addition to trying to generate random data, running some of these same properties on a large corpus of Python source code files. One obvious candidate is the standard library, but there's a lot of Python code out there we could potentially use. David MacIver used a corpus-based approach to test https://github.com/google/yapf/ with some efficacy: http://www.drmaciver.com/2015/03/27-bugs-in-24-hours/.

In general, I'd like to try push astroid testing toward defining what the code should do and then looking for falsifying examples in some automated fashion. I think we could also simplify some of the existing tests using this approach.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the rebuilder, to_string(), inference, and parent traversal named in the issue. Decide whether generated ASTs or a standard-library corpus is the initial input, then define a reproducible property test and document what a passing invariant or crash check demonstrates.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.