python / python/cpython

Replace module level mutable containers with immutable containers

未关闭
#139,003 5 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

performance stdlib topic-free-threading
主要语言
Python
星标
77.2k
派生
36k
PR 合并指标
PR 指标待抓取

描述

There are many module level containers with constants. By replacing the mutable containers with immutable variants (e.g. replace a set with a frozenset or a list with a tuple) we improve performance (especially in the free-threaded build, see for example https://github.com/python/cpython/pull/138429) and avoid accidental modification of these containers.

The number of module level lists, dicts and sets on current main is:

number of mutable module level containers by type:
<class 'dict'>: 266
<class 'list'>: 150
<class 'set'>: 63
<class 'collections.defaultdict'>: 2
<class '_strptime.TimeRE'>: 1
<class 'email._encoded_words._QByteMap'>: 1
Script to list all the module level mutable containers
import sys
import importlib
import pkgutil
from collections import Counter

blacklist = ('idlelib.idle')
excluded_submodule_names = ('__main__')
search_submodules = 2
mutable_containers = (list, dict,  set)

def list_container_types(module, mcc):
    print_module = False
    for a in dir(module):
        if a in ('__all__', '__path__', '__builtins__', '__annotations__', '__conditional_annotations__'):
            # why is __all__ a list and not a tuple?
            continue
        # if not a.startswith('_'):
        #    continue
        attr = getattr(module, a)
        tp = type(attr)
        if issubclass(tp, mutable_containers):
            if not print_module:
                print(f'{module}:')
                print_module = True
            print(f'  {a}: {tp}')
            mcc.update([tp])


def search_modules(module_names, search_submodules: int, mcc):
    for name in module_names:
        if name in blacklist:
            continue
        try:
            module = importlib.import_module(name)
        except:
            print(f'{name}: error on import')
            module = None
        list_container_types(module, mcc)

        if search_submodules:
            try:
                sub_names = list(z.name for z in pkgutil.iter_modules(module.__path__))
            except Exception as ex:
                sub_names = []
            mm = [name + '.' + sub_name for sub_name in sub_names if sub_name not in excluded_submodule_names]
            search_modules(mm, search_submodules - 1, mcc)

mcc = Counter()
module_names = sorted(list(sys.builtin_module_names)) + sorted(list(sys.stdlib_module_names))
search_modules(module_names, search_submodules=2, mcc = mcc)

print()
print('number of module level containers by type:')
for key, value in mcc.items():
    print(f'{key}: {value}')

Not all the mutable containers can be replaced by immutable containers. Some of them need to be mutable (e.g. copyreg.dispatch_table). And some of them are part of the public API and we might not want to change the type only for performance reasons.

Example candidates: _pydatetime._DAYNAMES (would improve performance of date.cdate), token.EXACT_TOKEN_TYPES, xml.etree.ElementTree.HTML_EMPTY

Linked PRs
  • gh-139004

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

该 issue 提供了一个模块扫描脚本,并列出了候选容器,包括 _pydatetime._DAYNAMES、token.EXACT_TOKEN_TYPES 和 xml.etree.ElementTree.HTML_EMPTY。首先查看链接的 PR gh-139004,并检查每个候选项的可变性以及公开 API 的使用情况。完成的标准是:在不改变所需行为的情况下,安全地转换合适的模块级容器。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
compilers
Issue 类型
重构
难度
5/5
预计耗时
一周以上
活跃度
停滞
描述清晰度
需要澄清
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。