Test stub files against real code

Start with the issue's proposed POC flow: build syntax trees for the source and stub, merge matching elements, and pass the enriched source to the existing type-checking mechanism. The first milestone is support for the listed source-only, stub-only, and matching-element cases, including interface mismatches; no specific files or tests are named.

There are a few use cases when stubs may be compelling over type annotations or type comments:

• annotate code that needs to be Python 2 compatible, ideally by using type annotations (so no type comments),
• often times users prefer to have type information outside of the source code (this is to avoid modifying legacy code change - and in practice lowers resistance from maintainers),
• the maintainer of the library does not want to take on the in-line type annotations, someone else trying to provide it as a separate package.

However at the moment this has some limitations: stub files are used only to check clients of the stub, not the code itself.

# a.py
def add_int(a, b):
    return a + b

add_int("a", "b") # mypy does not complain, actually ignores the entire file

# a.pyi
def add_int(a:int, b:int) -> int: ...

# client.py
from a import add_int
add_int(1, 2)
add_int(1, "2") # here mypy is going to complain

In case of the above files mypy will warn about bad usage of inside client.py, but not in a.py (that is, mypy does not validate against the source files being annotated by the stub). This has serious drawbacks:

• maintaining the stub files is hard, now users need to manually make sure whatever is in the stub file is an accurate representation of the matching source files (both interface - aka number of arguments, names - and type wise),
• the stub creator does not know if the source code itself is type correct or not.

Here I propose to implement a way to support testing stub files against the real code:

• build the syntax tree for the source file,
• build the syntax tree for the stub file,
• merge stub tree into the source tree
  • source file only tree available: dynamically typed - noop
  • abstract syntax tree only: raise error - missing annotated sources
  • both stub and source AST exists:
    • for all source elements for what there is a matching stub element
      • copy inject over type annotations
      • complain if function interface missmatch
• now just run the existing type checking mechanism on the enriched source code ast

Merging the AST definitely is not trivial. The plan is to start with a POC implementation, that works for the most cases, and then see the viability of this approach. Note this would help a lot both stub maintainers I think (typeshed) and projects that need Python 2 support.