python / python/cpython

tomllib: quadratic parse time from re-walking a deep table header for every key (related to gh-149231)

未关闭
#152,930 1 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

pending stdlib type-bug
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

Bug report

Bug description:

tomllib.loads re-traverses the whole table header path from the document root
once for every key/value pair under that header. A document with one table
header of depth D followed by M sibling keys is parsed in O(M * D) time.

key_value_rule (Lib/tomllib/_parser.py) computes
abs_key_parent = header + key_parent for each pair and then calls both
out.flags.is_(abs_key_parent, Flags.FROZEN) and
out.data.get_or_create_nest(abs_key_parent). Flags.is_ and
get_or_create_nest each walk the full abs_key_parent path from their
respective roots, so the shared header prefix is re-walked for every one of
the M keys. Nothing anchors the per-key work on the current table section.

Reproducer (single deep header, many single-part keys, all spec-valid):

import tomllib, time
D, M = 998, 80000
doc = "[" + ".".join(f"h{i}" for i in range(D)) + "]\n" + "".join(f"k{i}=1\n" for i in range(M))
t = time.perf_counter(); tomllib.loads(doc); print(len(doc), "bytes", time.perf_counter() - t, "s")

Measured (scaling D = M = L, so bytes are O(L) but time is O(L^2)):

L(=D=M)    bytes     time
   1000    11.5 KB   0.14 s
   2000    25.2 KB   0.57 s   (4.2x for 2x)
   4000    52.5 KB   2.19 s   (3.9x for 2x)

A single 52 KB file blocks the thread for over 2 seconds.

How this differs from gh-149231

gh-149231 fixed an O(N^2) blowup from a single key with N dotted parts, by
adding MAX_KEY_PARTS = sys.getrecursionlimit() in parse_key. This is a
different mechanism: the cost here comes from re-walking a shared header for
every one of M keys, and each key in the reproducer has exactly one part, so
the parse_key cap never fires on the keys. MAX_KEY_PARTS bounds the header
depth D to about 1000 but does not remove the per-key re-walk, so a residual
O(1000 * M) amplification survives on builds that carry the cap:

D=998, M=80000  ->  714 KB  ->  ~11.7 s   (legal even with the MAX_KEY_PARTS cap)
Suggested direction

Resolve the header's flags and nest node once per table section and have
key_value_rule walk only the relative key_parent (length 0 for the common
single-part key), turning each section from O(M * D) into O(M + D).

CPython versions tested on:

3.12, 3.13, 3.14, 3.15 (main). The full O(input^2) reproduces on 3.12 and on
3.14.5 as released (no MAX_KEY_PARTS yet); the O(1000 * M) residual reproduces
on 3.13, current 3.14, and main (which carry the gh-149231 cap).

Operating systems tested on:

macOS (also platform independent, pure Python parser)

Linked PRs
  • gh-152931

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

阅读 Lib/tomllib/parser.py,特别是 key_value_rule、parse_key、Flags.is 和 get_or_create_nest。运行复现程序以确认扩展性,然后比较更改前后在受支持的 Python 版本中的解析行为和性能。完成标准是,共享的表头处理不会针对每个键重复执行,同时有效 TOML 的解析仍然正确。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
performance
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
停滞
描述清晰度
描述清楚
新手友好度
35/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。