importlib does unnecessary os.stat calls checking for namespace __init__.py files

Overall, importlib is very good at caching and not searching over and over again. For instance, if you add new sys.path it won't traverse previous paths again. Except for one thing: it will check for the existence of __init__.py files on all namespace paths again.

It may sound minor, but on a project with heavy namespace use and a lot of sys.path it means a lot of extra FS access on each sys.path change. On windows it is especially slow.

Repro script:

import tempfile
import sys
import importlib
import os
from pathlib import Path

MODULES = [
    "namespace_1/foo.py",
    "namespace_1/bar.py",
    "namespace_1/xyz.py",
    "namespace_2/foo.py",
    "namespace_2/bar.py",
    "namespace_2/xyz.py",
]

with tempfile.TemporaryDirectory() as root:
    def _create_module(path):
        path = Path(root).joinpath(path)
        path.parent.mkdir(exist_ok=True)
        open(path, "w").close()

    for m in MODULES:
        _create_module(m)

    def my_stat(path):
        print(path)
        return os.stat(path)

    importlib._bootstrap_external._path_stat = my_stat
    sys.path.append(root)

    for i, m in enumerate(MODULES):
        # Critical line: invalidate sys.path cache
        sys.path.append(root + "/{i}")

        module_name = m.replace("/", ".").replace(".py", "")
        print(f"> importing: {module_name}")
        importlib.import_module(module_name)
     

After each import you will see again and again:

C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.cp37-win_amd64.pyd
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyd
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.py
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyw
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyc

Tested on python 3.7, but previously I checked it is the same on the latest one.

Inizia con lo script di riproduzione e ispeziona l'entry point importlib._bootstrap_external._path_stat di Python e il percorso di ricerca dei namespace package. Conferma i controlli ripetuti del filesystem dopo ogni modifica a sys.path, quindi aggiungi una copertura di regressione mirata. Il lavoro è completo quando gli import dei namespace rimangono corretti evitando al contempo i controlli ridondanti sull'esistenza di init.py.