Instagram / Instagram/LibCST

TypeInferenceProvider using another tool than pyre - Jedi

Open
#451 8 comments 3 reactions 0 assignees View on GitHub
codemod enhancement
Dominant language
Python
Stars
1.9k
Forks
229
PR merge metrics
No merged PRs in 30d

Description

[What follows is probably heresy given that Pyre is another Instagram project, so please don't "throw me to the pyre" :fire: - I mean no offence to that project]

I've found setting up a working pyre environment somewhat painful (building watchman from source, then getting a core dump because my project path was too long, etc.) and the documentation in LibCST of how to actually setup a ``TypeInferenceProvider`` and ``FullRepoManager`` to be lacking an example. Indeed, the best I've found was a screenshot of a notebook in https://github.com/Instagram/LibCST/pull/179, which I've diligently transformed into indexable form below (for my future self who wants to be able to google an example of doing it):

```
import libcst

from libcst.metadata.full_repo_manager import FullRepoManager
from libcst.metadata.type_inference_provider import TypeInferenceProvider

m = FullRepoManager('libCST/', ['_maybe_sentinel.py'], {TypeInferenceProvider})
for node, type_str in m.get_metadata_wrapper_for_path('_maybe_sentinel.py').resolve(TypeInferenceProvider).items():
code = libcst.parse_module("").code_for_node(node)
print(f'{code}: {type_str}')
```

As a result, I've also looked at other means of getting inference data given a node... [``Jedi's Script.infer``](https://jedi.readthedocs.io/en/latest/docs/api.html#jedi.Script.infer) is an interesting and seemingly simple option despite Jedi and LibCST working on a different level (a Script in Jedi has the ability to look through a virtual environment / PYTHONPATH to find references etc., much like Pyre and the ``FullRepoManager`` concept in LibCST).

A quick prototype later, I have a means to get hold of the Jedi inference for a node through a metadata provider (I think this is a testament to the LibCST code that this is so simple to do :+1:):

```
import libcst as cst
import libcst.metadata

import jedi

from typing import List

class TypeInferenceFromJediProvider(cst.BatchableMetadataProvider[List[jedi.api.classes.Name]]):
METADATA_DEPENDENCIES = (libcst.metadata.PositionProvider, )
gen_cache = True # We need the cache to contain a jedi.Script.

def __init__(self, cache) -> None:
super().__init__(cache)
self._script: jedi.Script = self.cache['script']

def _parse_metadata(self, node: cst.CSTNode) -> None:
pos = self.get_metadata(libcst.metadata.PositionProvider, node).start
self.set_metadata(node, self._script.infer(pos.line, pos.column))

def visit_Name(self, node: cst.Name):
self._parse_metadata(node)

def visit_Attribute(self, node: cst.Attribute):
self._parse_metadata(node)

def visit_Call(self, node: cst.Call):
self._parse_metadata(node)
```

Which is used as:

```

class InferencePrinter(cst.CSTVisitor):
METADATA_DEPENDENCIES = (cst.metadata.PositionProvider, TypeInferenceFromJediProvider)

def visit_Name(self, node: cst.Name) -> None:
pos = self.get_metadata(cst.metadata.PositionProvider, node).start
possible_types: List[jedi.api.classes.Name] = self.get_metadata(TypeInferenceFromJediProvider, node)
print(f"{node.value} found at line {pos.line}, column {pos.column}")
print(f"Source: {', '.join(possible_type.full_name for possible_type in possible_types)}")
print()

prj = jedi.Project('example-project')
code = '''
from collections import namedtuple

class MyThing(namedtuple('MyThing', ['a'])):
pass

thing = MyThing()

thing.__len__
'''
script = jedi.Script(code, project=prj)

module = cst.parse_module(code)
wrapper = cst.metadata.MetadataWrapper(module, cache={TypeInferenceFromJediProvider: {'script': script}})

wrapper.visit(InferencePrinter())
```

Output:

```
collections found at line 2, column 5
Source: collections

namedtuple found at line 2, column 24
Source: collections.namedtuple

MyThing found at line 4, column 6
Source: __main__.MyThing

namedtuple found at line 4, column 14
Source: collections.namedtuple

thing found at line 7, column 0
Source: __main__.MyThing

MyThing found at line 7, column 8
Source: __main__.MyThing

thing found at line 9, column 0
Source: __main__.MyThing

__len__ found at line 9, column 6
Source: builtins.tuple.__len__
```

Given the knowledge that Jedi and LibCST are both using parso under the hood, I didn't look into trying to avoid multiple parsing stages (performance isn't so critical to me, and the performance was good enough).

I just wanted to write this down so that others can benefit - I don't expect this will make its way into the LibCST codebase. (please feel free to close the issue!)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.