python / python/cpython

Improve performance of `re.compile()` for patterns with character ranges

未关闭
#149,427 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

performance stdlib topic-regex type-feature
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Feature or enhancement

Proposal:

re._compiler._optimize_charset builds a 256-byte (or wider) charmap for character classes. On main it is filled with a loop:

for i in range(av[0], av[1] + 1):
    charmap[i] = 1

Replacing the loop with a single bytearray slice is faster.

Benchmark results on pyperformance regex_compile

main (86.2 ± 1.4 ms)  ->  PR (78.5 ± 1.6 ms):  1.10x faster

Microbenchmarks:

compile-charsets-small:  2.80 ± 0.04 ms  ->  1.22 ± 0.02 ms:  2.29x faster
compile-charsets-big:    1.46 ± 0.03 ms  ->  1.36 ± 0.03 ms:  1.08x faster
Geometric mean:                                              1.57x faster

The "small" case oversamples wide ranges ([\x00-\xff], BMP range) which benefit most. The "big" case has mostly small ASCII ranges where the slice-fill construction (b'\x01' * n) competes with the loop savings.

microbench script
"""Microbench for the bytearray slice-fill optimization in
re._compiler._optimize_charset (RANGE handler)."""
import pyperf
import re

SMALL_PATTERNS = [
    r'[a-zA-Z0-9_]+',
    r'[^\s\d]+',
    r'[a-z][A-Z][0-9]',
    r'[\x00-\xff]',                  # full byte range
    r'[Ā-῿]+',                       # BMP range, hits growth path
    r'[a-zA-Z0-9!@#$%^&*()_+\-=\[\]{};:\'",.<>/?\\|`~]+',
    r'[0-9a-fA-F]{2,8}',
    r'[ -~]+',                       # all printable ASCII
]

BIG_PATTERN = (
    r'^([a-zA-Z][a-zA-Z0-9_\-]*)\s*'
    r'([\x20-\x7e]*)\s*'
    r'([-ÿĀ-ɏ]+)?\s*'
    r'([0-9]{1,4}[\-/.][0-9]{1,2}[\-/.][0-9]{1,4})?\s*'
    r'([^\s,;:]+)$'
)

def compile_small():
    for _ in range(8):
        for p in SMALL_PATTERNS:
            re.purge()
            re.compile(p)

def compile_big():
    for _ in range(10):
        re.purge()
        re.compile(BIG_PATTERN)

runner = pyperf.Runner()
runner.bench_func('compile-charsets-small', compile_small)
runner.bench_func('compile-charsets-big', compile_big)
Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

Linked PRs
  • gh-149428

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

从 re._compiler._optimize_charset 开始,重点查看填充 charmap 的 RANGE 处理器。检查关联的 PR gh-149428,并比较其对小字符范围和宽字符范围的处理行为。完成的标准是范围处理仍然正确,并且 regex_compile 和提供的微基准测试能够复现预期的性能改进。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
performance
Issue 类型
功能
难度
2/5
预计耗时
1-3 小时
活跃度
停滞
描述清晰度
描述清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。