Circular import error for `threading` when using `importlib.util.LazyLoader` with a custom finder
まだ誰も着手していません。
- 主要言語
- Python
- スター
- 77.2k
- フォーク
- 35.9k
- PR マージ指標
- PR 指標を取得中
説明
Bug report
Bug description:
When using a custom finder (on sys.meta_path) that wraps a module spec's loader with importlib.util.LazyLoader, the threading import within importlib.util.LazyLoader.exec_module is attempted with the custom finder, causing a circular import error.
Not sure if this is more of a feature request than a bug report, but the behavior was certainly surprising at first glance. Hopefully there's a way to better support this use case from within the stdlib without too much burden. Otherwise, users of LazyLoader in this manner would be forced to either a) maintain an exclusion list of modules that their finder ignores, potentially including all of threading's dependencies or b) always import threading before using such finders, which shouldn't be necessary, especially if threading isn't directly used by the user code.
Reproducer
import importlib.util
import sys
class LazyFinder:
"""A module spec finder that wraps a spec's loader, if it exists, with LazyLoader."""
@classmethod
def find_spec(cls, fullname: str, path=None, target=None, /):
for finder in sys.meta_path:
if finder is not cls:
spec = finder.find_spec(fullname, path, target)
if spec is not None:
break
else:
raise ModuleNotFoundError(...)
if spec.loader is not None:
spec.loader = importlib.util.LazyLoader(spec.loader)
return spec
class LazyFinderContext:
"""Temporarily "lazify" some types of import statements in the runtime context."""
def __enter__(self):
if LazyFinder not in sys.meta_path:
sys.meta_path.insert(0, LazyFinder)
def __exit__(self, *exc_info):
try:
sys.meta_path.remove(LazyFinder)
except ValueError:
pass
with LazyFinderContext():
import inspect
Expected Output
No error.
Actual Output
> python scratch.py
Traceback (most recent call last):
File "/home/thanos/projects/personal/pycc/scratch.py", line 40, in <module>
import inspect
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib.util>", line 257, in exec_module
File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
File "<frozen importlib.util>", line 267, in exec_module
AttributeError: partially initialized module 'threading' has no attribute 'RLock' (most likely due to a circular import)
Commentary
This is technically caused by this code:
https://github.com/python/cpython/blob/9dabace39d118ec7a204b6970f8a3f475a11522c/Lib/importlib/util.py#L256-L260
However, it is based on the fair assumptions that LazyLoader a) isn't critical to CPython startup, and b) won't be used in a circular fashion with a custom finder. However, there are use cases for such a finder (e.g. https://github.com/scientific-python/lazy-loader/pull/121#issuecomment-2457961532, one place this issue was discovered). While finders utilizing the lazy loader could work around this with an exclusion list of modules (e.g. mercurial's lazy loader does), I think users would find using LazyLoader easier with the finder-related import hooks if that wasn't necessary.
Based on the commit history, a top-level import for threading breaks gevent, so I'd rather not repeat that. I'm not very familiar with gevent, but if using _thread.RLock is fine in importlib._bootstrap for the module locks, then maybe that could be used in importlib.util as well?
EDIT: Updated code snippet to be consistent with output; accidentally used a different import while testing. Still, same result.
CPython versions tested on:
3.12, 3.13
Operating systems tested on:
Linux, Windows
コントリビューションガイド
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
調査の方向性
まず、提供された再現プログラムをカスタム finder で実行し、Lib/importlib/util.py を調べます。特に LazyLoader.exec_module と、256-260 前後にリンクされている行を確認してください。Python 3.12 と 3.13 での動作を比較します。再現プログラムが、指定された起動時の制約を維持したまま、循環インポートによる AttributeError なしで inspect をインポートできれば完了です。
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- backend, tooling
- issue の種類
- バグ
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 活発さ
- 停滞
- 明瞭さ
- おおむね明確
- 初心者へのやさしさ
- 35/100