python / python/cpython

Circular import error for `threading` when using `importlib.util.LazyLoader` with a custom finder

未关闭
#127,116 0 条评论 1 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

topic-importlib type-bug
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

When using a custom finder (on sys.meta_path) that wraps a module spec's loader with importlib.util.LazyLoader, the threading import within importlib.util.LazyLoader.exec_module is attempted with the custom finder, causing a circular import error.

Not sure if this is more of a feature request than a bug report, but the behavior was certainly surprising at first glance. Hopefully there's a way to better support this use case from within the stdlib without too much burden. Otherwise, users of LazyLoader in this manner would be forced to either a) maintain an exclusion list of modules that their finder ignores, potentially including all of threading's dependencies or b) always import threading before using such finders, which shouldn't be necessary, especially if threading isn't directly used by the user code.

Reproducer
import importlib.util
import sys

class LazyFinder:
    """A module spec finder that wraps a spec's loader, if it exists, with LazyLoader."""

    @classmethod
    def find_spec(cls, fullname: str, path=None, target=None, /):
        for finder in sys.meta_path:
            if finder is not cls:
                spec = finder.find_spec(fullname, path, target)
                if spec is not None:
                    break
        else:
            raise ModuleNotFoundError(...)

        if spec.loader is not None:
            spec.loader = importlib.util.LazyLoader(spec.loader)

        return spec


class LazyFinderContext:
    """Temporarily "lazify" some types of import statements in the runtime context."""
    
    def __enter__(self):
        if LazyFinder not in sys.meta_path:
            sys.meta_path.insert(0, LazyFinder)

    def __exit__(self, *exc_info):
        try:
            sys.meta_path.remove(LazyFinder)
        except ValueError:
            pass

with LazyFinderContext():
    import inspect
Expected Output

No error.

Actual Output
> python scratch.py
Traceback (most recent call last):
  File "/home/thanos/projects/personal/pycc/scratch.py", line 40, in <module>
    import inspect
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib.util>", line 257, in exec_module
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib.util>", line 267, in exec_module
AttributeError: partially initialized module 'threading' has no attribute 'RLock' (most likely due to a circular import)
Commentary

This is technically caused by this code:
https://github.com/python/cpython/blob/9dabace39d118ec7a204b6970f8a3f475a11522c/Lib/importlib/util.py#L256-L260

However, it is based on the fair assumptions that LazyLoader a) isn't critical to CPython startup, and b) won't be used in a circular fashion with a custom finder. However, there are use cases for such a finder (e.g. https://github.com/scientific-python/lazy-loader/pull/121#issuecomment-2457961532, one place this issue was discovered). While finders utilizing the lazy loader could work around this with an exclusion list of modules (e.g. mercurial's lazy loader does), I think users would find using LazyLoader easier with the finder-related import hooks if that wasn't necessary.

Based on the commit history, a top-level import for threading breaks gevent, so I'd rather not repeat that. I'm not very familiar with gevent, but if using _thread.RLock is fine in importlib._bootstrap for the module locks, then maybe that could be used in importlib.util as well?

EDIT: Updated code snippet to be consistent with output; accidentally used a different import while testing. Still, same result.

CPython versions tested on:

3.12, 3.13

Operating systems tested on:

Linux, Windows

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先,使用自定义 finder 运行提供的复现程序,并检查 Lib/importlib/util.py,尤其是 LazyLoader.exec_module 以及 256-260 附近链接的行。在 Python 3.12 和 3.13 上比较行为;当复现程序能够在保持所述启动约束的同时导入 inspect,且不出现循环导入导致的 AttributeError 时,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
backend, tooling
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。