python / python/cpython

Reading gzip file with very long filename or comment takes long time

未关闭
#150,144 2 条评论 2 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

3.13 3.14 3.15 performance stdlib type-bug
主要语言
Python
星标
77.2k
派生
35.9k
PR 合并指标
PR 指标待抓取

描述

The gzip file can contain filename and comment which are written like null terminated sequences of bytes. GzipFile ignores filename and comment (only calculates their checksum if needed), but simply searching for the terminating null byte, while reading byte-by-byte, takes time. On my computer, with fast CPU and SSD, reading a gzip file containing 1 GiB filename or comment will take over 5 minutes. This is not a security issue per se, because to trigger it, attacker need to send a large file at first place, but this is not fine.

This issue was discovered during discussion in #149945. The original proposed solution for that issue imposed a limit on the size of filename and comment. While the limit on filename is reasonable (but it can depend on platform?), we cannot be sure that there are no uses cases for large comments.

The following PR uses reading by chunks of growing size. It reads a 1 GiB header in fractions of second.

Linked PRs
  • gh-150145

贡献指南

打开贡献指南

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

调研方向

首先检查 GzipFile 对以 null 结尾的文件名字段和注释字段的处理方式,然后比较 PR #150145 中提出的分块读取方案。当读取包含超长字段的 header 不再花费过多时间逐字节扫描时,即可视为完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
backend
Issue 类型
缺陷
难度
3/5
预计耗时
1-2 天
活跃度
停滞
描述清晰度
基本清楚
新手友好度
25/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。