pylint-dev / pylint-dev/astroid

error upon `import_module` in context where package and module exists with the same name

Open
#981 1 comment 0 reactions 0 assignees View on GitHub
Discussion 🤔 Enhancement ✨ Help wanted 🙏
Dominant language
Python
Stars
582
Forks
357
Avg merge
1d 1h
Merged PRs (30d)
23

Description

I'm not sure whether this is considered a bug, but even if considered an "enhancement", it'd be very good to address the situation I'm going to describe.

1. Within a Python script, a given name may identify either a package or a module.
2. Despite of item 1, a Python codebase/project may still `import`, in different scripts (or contexts), a package or a module under the same name.
3. But, in a scenario such as that described in item 2, astroid's `import_module(…)` [function](https://astroid.readthedocs.io/en/latest/api/astroid.nodes.html#astroid.nodes.Module.import_module) will always fail for at least of the same-named package or module `import`.

This is a project to illustrate the problem.

Screen Shot 2021-04-29 at 16 29 17

- script1.py
```
import sys
import os.path

base_dir = os.path.dirname(os.path.realpath(__file__))
lib_dir = os.path.join(base_dir, 'main-site-packages')
sys.path.append(lib_dir)

from nameofsomething.pack.lib import f

print('script 1')
```

- script2.py
```
import sys
import os.path

base_dir = os.path.dirname(os.path.realpath(__file__))
lib_dir = os.path.join(base_dir, 'extra-site-packages')
sys.path.append(lib_dir)

from nameofsomething import g

print('script 2')
```

Both scripts can be run independently, without any error. Even though, _script1.py_ imports the **module** `nameofsomething` and _script2.py_ imports the **package** `nameofsomething`.

Now, let's make the (reasonable fair, but not strictly correct) assumption that my astroid-based tool scans all files inside _basedir_ for analysis… it traverses the AST of both _script1.py_ and _script2.py_ and, upon visitation of an `import`, it attempts to load the package/module in question. Below is simplified/emulated version of the tool's behaviour, with only the relevant parts:

- astroidfail.py
```
import sys
import os.path
from astroid.manager import AstroidManager

def fail():
M = AstroidManager()
base_dir = os.path.dirname(os.path.realpath(__file__))

extra_lib_dir = os.path.join(base_dir, 'extra-site-packages')
sys.path.append(extra_lib_dir)
script2_path = os.path.join(base_dir, 'script2.py')
node = M.ast_from_file(script2_path)
node.import_module('nameofsomething')

main_lib_dir = os.path.join(base_dir, 'main-site-packages')
sys.path.append(main_lib_dir)
script1_path = os.path.join(base_dir, 'script1.py')
node = M.ast_from_file(script1_path)
node.import_module('nameofsomething.pack.lib')

if __name__ == '__main__':
fail()
```

Running this code, the following error is triggered:

Screen Shot 2021-04-29 at 16 34 52

This error makes sense, since the `sys.path` contains both a package and a module under the same name, i.e., `nameofsomething`, violating item 1 from my list at the top. One could keep separate managers or (somehow) isolate the processing of each script, but this will typically lead to more expensive computation. So (I believe) that this situation is common among tools that perform static analysis over a codebase/project as a whole, as in item 2 from my list.

Can we get this fixed?

From what I can tell by inspection/debugging the problem is in [the _find_spec_with_path function](https://github.com/PyCQA/astroid/blob/b964e1f5e04194221b795594d3924b7eb3e8e312/astroid/interpreter/_import/spec.py#L326-L334) and [its successive invocations](https://github.com/PyCQA/astroid/blob/b964e1f5e04194221b795594d3924b7eb3e8e312/astroid/interpreter/_import/spec.py#L363-L368). Note that different finders are used to locate a module, in particular [the ImportlibFinder](https://github.com/PyCQA/astroid/blob/b964e1f5e04194221b795594d3924b7eb3e8e312/astroid/interpreter/_import/spec.py#L244) and [the PathSpecFinder](https://github.com/PyCQA/astroid/blob/b964e1f5e04194221b795594d3924b7eb3e8e312/astroid/interpreter/_import/spec.py#L246); however, the module name is search by parts `part1.part2…partN`, and, whenever one of its subparts has match, it's considered to be the right one. In the situation I describe, though, this reasoning is incorrect, because for the `import` of **package** `nameofsomething.pack.lib` we may end up first finding the **module** `nameofsomething.py` and flagging it as the correct match, but it isn't… leading to subsequent failure when the part `pack` of the package is searched for. I suppose that iterating over the spec finders and stopping the search once the entire **full name** is matched would fix the issue.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with astroid/interpreter/_import/spec.py, especially _find_spec_with_path and its successive invocations, then reproduce the issue with the provided script1.py, script2.py, and astroidfail.py example. Compare the ImportlibFinder and PathSpecFinder paths while resolving nameofsomething.pack.lib and nameofsomething; done means both same-named package and module imports resolve in their respective contexts without the reported failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.