python / python/cpython

Speed up matching of case-insensitive character sets

未关闭
#152,054 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

performance stdlib topic-regex type-feature
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Feature or enhancement

Proposal:

A REPEAT_ONE over a case-insensitive character set — e.g. [a-z]+ with re.IGNORECASE — does not use the fast SRE(count) path. The compiled inner opcode is IN_IGNORE / IN_UNI_IGNORE / IN_LOC_IGNORE, none of which has a case in SRE(count). The case-sensitive SRE_OP_IN already has a fast case.

Adding the three IN_*_IGNORE cases to SRE(count) lets them scan inline.

Benchmark

Benchmark before after
[a-z]+ re.I|re.A (IN_IGNORE) 1.28 us 538 ns 2.38x
[a-z]+ re.I (IN_UNI_IGNORE) 1.41 us 711 ns 1.98x
[aeiou]+ re.I 1.35 us 706 ns 1.91x
[a-z0-9]+ re.I 1.31 us 696 ns 1.88x
[a-z0-9_]+ re.I 1.31 us 703 ns 1.86x
[a-z]+ re.L|re.I bytes (IN_LOC_IGNORE) 2.09 us 1.38 us 1.52x
findall [a-z]+ re.I 109 us 88.8 us 1.22x
findall [a-z_][a-z0-9_]* re.I 103 us 89.2 us 1.15x

[^0-9]+ re.I is unchanged — it has no cased members, so it stays a plain
IN (already fast).

benchmark script (pyperf)
"""Benchmark: SRE(count) fast path for case-insensitive set repeats."""
import re
import pyperf

N = 100
MIXED   = ("aBcDeFgHiJkLmNoPqRsTuVwX" * N)[:N]
ALNUM   = ("aB3dE6gH9kLmN0pQrStUvWx1" * N)[:N]
WORD    = ("aB_dE_gH_kLmN_pQrStUvW_1" * N)[:N]
NODIGIT = ("aBcDeF gHiJkL!mNoPqR.sT?" * N)[:N]
BYTES   = MIXED.encode("latin1")

SCANS = [
    ("scan_alpha_uni",  re.compile(r"[a-z]+",    re.I),        MIXED),
    ("scan_alpha_asc",  re.compile(r"[a-z]+",    re.I | re.A), MIXED),
    ("scan_alnum_uni",  re.compile(r"[a-z0-9]+", re.I),        ALNUM),
    ("scan_word_uni",   re.compile(r"[a-z0-9_]+",re.I),        WORD),
    ("scan_neg_uni",    re.compile(r"[^0-9]+",   re.I),        NODIGIT),
    ("scan_vowels_uni", re.compile(r"[aeiou]+",  re.I),        "aAeEiIoOuU" * (N // 10)),
    ("scan_alpha_loc",  re.compile(rb"[a-z]+",   re.L | re.I), BYTES),
]
DOC = ("The Quick Brown Fox jumps over 12 Lazy Dogs near IP 10_0_0_1 and Node7. " * 50)
FINDS = [
    ("find_words_ci", re.compile(r"[a-z]+",           re.I), DOC),
    ("find_ident_ci", re.compile(r"[a-z_][a-z0-9_]*", re.I), DOC),
]

def make_scan(p, s):
    def run():
        assert p.match(s) is not None
    return run

runner = pyperf.Runner()
for name, p, s in SCANS:
    runner.bench_func(name, make_scan(p, s))
for name, p, s in FINDS:
    runner.bench_func(name, (lambda p, s: lambda: p.findall(s))(p, s))

Run under the unpatched and patched builds, then
python -m pyperf compare_to before.json after.json --table.

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs
  • gh-152055

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先检查链接的 PR gh-152055,然后检查 SRE(count) 入口点以及 IN、IN_IGNORE、IN_UNI_IGNORE 和 IN_LOC_IGNORE opcode 的处理。使用未打补丁和已打补丁的构建运行提供的 pyperf 基准测试,并确认不区分大小写的字符集重复基准测试有所改善,同时不改变保持不变的否定集情况。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
performance
Issue 类型
功能
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。