python / python/mypy

Mypy isn't discovering all modules in namespace packages with multiple __path__ components

Open
#16,683 1 comment 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Bug Report

It seems that mypy isn't considering all parts of namespace package paths, and is only checking the sources corresponding to the first path in its path.

Hopefully the example below clarifies it. I searched quite a bit in the bug tracker/docs, but didn't find anything describing my scenario, so apologies if it's a dupe!

To Reproduce

Consider the following hierarchy

./pkg_a
./pkg_a/foo
./pkg_a/foo/module1.py
./pkg_a/foo/py.typed
./pkg_b
./pkg_b/foo
./pkg_b/foo/py.typed
./pkg_b/foo/module2.py

foo.module1 and foo.module2 simply contain a bit of code to demonstrate mypy error detection:

$ cat ./pkg_a/foo/module1.py
trigger_mypy_error: str = 123
$ cat ./pkg_b/foo/module2.py
trigger_mypy_error: str = 123

Just as expected, we have a namespace package, both modules are imported without issues:

$ PYTHONPATH=pkg_b:pkg_a python3 -c 'import foo; print(foo.__path__); import foo.module1; import foo.module2'
_NamespacePath(['/project/for_mypy_bugtracker/pkg_b/foo', '/project/for_mypy_bugtracker/pkg_a/foo'])

Actual Behavior

When type checking, mypy only detects module2.py and misses module1.py:

$ MYPYPATH=pkg_b:pkg_a mypy --namespace-packages --strict -p foo
pkg_b/foo/module2.py:1: error: Incompatible types in assignment (expression has type "int", variable has type "str")  [assignment]
    trigger_mypy_error: str = 123
                              ^~~
Found 1 error in 1 file (checked 2 source files)

Seems that the only sources detected are from pkg_b:

MYPYPATH=pkg_b:pkg_a mypy --namespace-packages -p foo -v 2>&1 | grep BuildSource
LOG:  Found source:           BuildSource(path='pkg_b/foo', module='foo', has_text=False, base_dir=None, followed=False)
LOG:  Found source:           BuildSource(path='pkg_b/foo/module2.py', module='foo.module2', has_text=False, base_dir=None, followed=False)

I dug a bit and I suspect it has something to do with this line
https://github.com/python/mypy/blob/1dd8e7fe654991b01bd80ef7f1f675d9e3910c3a/mypy/modulefinder.py#L559 -- the result of finding the module is just one path (pkg_b in this case), so it doesn't have a chance to explore pkg_a.

Expected Behavior

Mypy should detect both foo.module1 and foo.module2

Your Environment

  • Mypy version used: 1.7.1 (latest)
  • Python version used: 3.10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in mypy/modulefinder.py around the referenced line, then reproduce the issue with the pkg_a/pkg_b hierarchy and the MYPYPATH=pkg_b:pkg_a mypy --namespace-packages --strict -p foo command. Done means source discovery includes both namespace-package components and type checking reports errors from module1.py and module2.py.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.