Improve `tarfile` streaming mode to handle very large archives
还没有人认领这个 Issue。
- 主要语言
- Python
- 星标
- 77.2k
- 派生
- 35.9k
- PR 合并指标
- PR 指标待抓取
描述
Bug report
Bug description:
I am trying to use tarfile to write very large archives, and the process is being killed by the OOM killer. I would expect+want to be able to use it in streaming mode (write an unlimited number of files) like the standard tar command line utility supports by default.
Reproduction case attached.
import gc
import io
import os
import psutil
import tarfile
if __name__ == "__main__":
t = tarfile.open("a.tar", mode="w") # default compresslevel 9
for i in range(1,100_000_000):
if i % 10_000 == 0:
gc.collect()
process = psutil.Process(os.getpid())
mem_info = process.memory_info()
mem = mem_info.rss
print(f"Iteration {i}, memory usage: {mem}")
bs = (" "*1000 + str(i)).encode('utf8')
with io.BytesIO(bs) as file:
tarinfo = tarfile.TarInfo(name="cool_files/{i}.txt")
tarinfo.size = len(bs)
t.addfile(tarinfo, file)
The memory usage increases without bound because the list of this line in addfile():
self.members.append(tarinfo)
I'm not sure what the use-case is for this line. In write-only mode, it does not seem useful. Maybe mixed read/write? But in general it does not seem correct to assume you can fit all the tarinfo's in memory.
Edit: As a workaround, I'm setting t.members=[] manually.
CPython versions tested on:
3.13
Operating systems tested on:
Linux
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
调研方向
从 tarfile 中 TarFile.addfile() 附近的实现以及 self.members.append(tarinfo) 这一行开始,使用报告中的复现来观察内存增长。确定 write-only 或 streaming 模式的预期行为,然后验证写入超大归档时内存使用量能够保持在有界范围内,同时不会破坏 tarfile 的其他使用场景。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- python
- 领域
- backend
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 停滞
- 描述清晰度
- 基本清楚
- 新手友好度
- 45/100