importlib does unnecessary os.stat calls checking for namespace __init__.py files

Overall, importlib is very good at caching and not searching over and over again. For instance, if you add new sys.path it won't traverse previous paths again. Except for one thing: it will check for the existence of __init__.py files on all namespace paths again.

It may sound minor, but on a project with heavy namespace use and a lot of sys.path it means a lot of extra FS access on each sys.path change. On windows it is especially slow.

Repro script:

import tempfile
import sys
import importlib
import os
from pathlib import Path

MODULES = [
    "namespace_1/foo.py",
    "namespace_1/bar.py",
    "namespace_1/xyz.py",
    "namespace_2/foo.py",
    "namespace_2/bar.py",
    "namespace_2/xyz.py",
]

with tempfile.TemporaryDirectory() as root:
    def _create_module(path):
        path = Path(root).joinpath(path)
        path.parent.mkdir(exist_ok=True)
        open(path, "w").close()

    for m in MODULES:
        _create_module(m)

    def my_stat(path):
        print(path)
        return os.stat(path)

    importlib._bootstrap_external._path_stat = my_stat
    sys.path.append(root)

    for i, m in enumerate(MODULES):
        # Critical line: invalidate sys.path cache
        sys.path.append(root + "/{i}")

        module_name = m.replace("/", ".").replace(".py", "")
        print(f"> importing: {module_name}")
        importlib.import_module(module_name)
     

After each import you will see again and again:

C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.cp37-win_amd64.pyd
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyd
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.py
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyw
C:\Users\[user]\AppData\Local\Temp\tmpz9rexobt\namespace_2\__init__.pyc

Tested on python 3.7, but previously I checked it is the same on the latest one.

從重現腳本開始，檢查 Python 的 importlib._bootstrap_external._path_stat 進入點和命名空間套件查找路徑。確認每次修改 sys.path 後都會重複進行檔案系統檢查，然後加入有針對性的回歸覆蓋。完成的標準是命名空間匯入仍然正確，同時避免對 init.py 是否存在進行多餘的檢查。