python / python/mypy

Disallow implicit exports on implicit submodule name bindings in `__init__.py`s

Open
#11,024 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug topic-implicit-reexport
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Consider the following package and main.py:

.
├── main.py
└── pkg
    ├── __init__.py
    ├── module1.py
    ├── module2.py
    └── utils.py

main.py:

import pkg
pkg.utils.bar1(pkg.foo1())

__init__.py:

__all__ = ('foo1', 'foo2')

from .module1 import foo1
from .module2 import foo2

module1.py:

def foo1() -> int:
    num = 1
    return num

module2.py:

from .utils import bar2

def foo2() -> str:
    num = 2
    return bar2(num)

utils.py:

def bar1(num: int) -> str:
    return 'a'

def bar2(num: int) -> str:
    return 'b'

There are no errors with any of this code; running main.py works fine, type checking with mypy --strict main.py or mypy --strict my_package checks out; but the package arguably contains an inaccuracy. Because of the way the package is written, the consumer script main.py relies on a potentially unstable language detail that is not widely known, although a documented part of the language:

When a submodule is loaded using any mechanism a binding is placed in the parent module’s namespace to the submodule object.

Let’s say that foo2 is deprecated and module2 is not needed any more. If the import line from .module2 import foo2 is removed from __init__.py then nothing would import the pkg.utils module anymore, hence the utils attribute will no longer exist on pkg. This means pkg.utils will no longer work in main.py without an explicit import pkg.utils.

This behaviour is indeed surprising and I wish Python didn’t do this implicit submodule name binding, although I acknowledge this feature has important implications as to how import foo.bar.baz currently behaves. However, I’m sure we can agree that having from .foo import bar leak foo into the namespace inside __init__.py is a common unintended side effect of importing bar.

I think it would be beneficial if Mypy could enforce a new convention to detect and disallow imports of names that exist in __init__.py due to implicit submodule name binding.

For comparison, Pyright does not understand implicit submodule name binding and simply emits a "utils" is not a known member of module in the given scenario.

Expected Behavior
main.py:3: error: Module "pkg" does not explicitly export submodule "utils"; implicit submodule export disabled
Actual Behavior
Success: no issues found in 1 source file
My Environment
  • Mypy version used: 0.910
  • Python version used: 3.9.6
  • Pyright version used: 1.1.163

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the package example in init.py and main.py, then reproduce the current result with mypy --strict main.py. Define how explicit exports should be distinguished from implicit submodule bindings; done means the example reports that pkg does not explicitly export utils while preserving the documented import behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.