importlib does unnecessary os.stat calls checking for namespace __init__.py files

Overall, importlib is very good at caching and not searching over and over again. For instance, if you add new sys.path it won't traverse previous paths again. Except for one thing: it will check for the existence of __init__.py files on all namespace paths again.

It may sound minor, but on a project with heavy namespace use and a lot of sys.path it means a lot of extra FS access on each sys.path change. On windows it is especially slow.

Repro script:

import tempfile
import sys
import importlib
import os
from pathlib import Path

MODULES = [
    "namespace_1/foo.py",
    "namespace_1/bar.py",
    "namespace_1/xyz.py",
    "namespace_2/foo.py",
    "namespace_2/bar.py",
    "namespace_2/xyz.py",
]

with tempfile.TemporaryDirectory() as root:
    def _create_module(path):
        path = Path(root).joinpath(path)
        path.parent.mkdir(exist_ok=True)
        open(path, "w").close()

    for m in MODULES:
        _create_module(m)

    def my_stat(path):
        print(path)
        return os.stat(path)

    importlib._bootstrap_external._path_stat = my_stat
    sys.path.append(root)

    for i, m in enumerate(MODULES):
        # Critical line: invalidate sys.path cache
        sys.path.append(root + "/{i}")

        module_name = m.replace("/", ".").replace(".py", "")
        print(f"> importing: {module_name}")
        importlib.import_module(module_name)
     

After each import you will see again and again:

C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.cp37-win_amd64.pyd
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyd
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.py
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyw
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyc

Tested on python 3.7, but previously I checked it is the same on the latest one.

Comece pelo script de reprodução e inspecione o ponto de entrada importlib._bootstrap_external._path_stat do Python e o caminho de busca de namespace packages. Confirme as verificações repetidas do sistema de arquivos após cada alteração em sys.path e, em seguida, adicione uma cobertura de regressão direcionada. O trabalho estará concluído quando as importações de namespace continuarem corretas, evitando verificações redundantes da existência de init.py.