Rewrite .test as .py

Many of us are using editors that handle Python code (syntax highlighting, symbol browsing). Would it be a good idea to restructure the .test data-files as Python files?

I've been musing about it for a while, so hear me out, then let me know if it's a stupid idea :)

Example:

-[case testFoo]
-s1: str = 42  # E: Incompatible types in assignment (expression has type "int", variable has type "str")
+def test_foo():
+  s1: str = 42  # E: Incompatible types in assignment (expression has type "int", variable has type "str")

To clarify, this wouldn't be true Python code, i.e. it won't be fed directly to CPython or Mypy; rather, it'll be a Python-flavored data file — perhaps parsed by AST (if feasible and performant), but where the function bodies are passed to the current machinery as text.

Extra qualifies like # flags: --strict-optional could be represented as decorators, e.g.

@flags('--strict-optional')
def test_foo():
   ...

[case testFoo-xfail] would become:

@pytest.mark.xfail
def test_foo():
   ...

[out] would become:

@expected_output("""
main:1: Incompatible types in assignment (expression has type "int", variable has type "str")
""")
def test_foo():
   ...

[file a.py] would become:

@testcase
def test_foo():
   ...  # main goes here


@test_foo.file('a.py')
def test_foo():
   ...  # a.py goes here

Benefits:

• First-class support in IDEs, e.g.
  
  
• The disarray of testcase options, sections and modifiers could be formalized in a Pythonic syntax and a module that would document them, e.g. from mypy.testcase import expected_output and then

  # mypy/testcase.py
  
  def expected_output(output: str) -> Testcase:
      """Documentation goes here"""
      ... 
  

• Can use some Python tooling like Black
• A more flexible representation than INI, while providing first-class editor experience (unlike, say, converting to YAML and putting the Python test cases into a multiline YAML block).
• More natural for newcomers?

Cons:

• Harder to implement?
• Confusing that it's Python but not really?
• Less natural to newcomers?

Begin with the existing .test data-file format and the current machinery that consumes testcase bodies; compare that path with parsing the proposed Python-flavored files via AST. A useful outcome is a decision on feasibility, syntax for options, sections and modifiers, and whether the format can preserve existing test behavior.