tarfile: unbounded memory use on large pax and GNU extensions
還沒有人認領這個 Issue。
- 主要語言
- Python
- 星號
- 77.2k
- 分支
- 36k
- PR 合併指標
- PR 指標待擷取
描述
Bug report
Bug description:
This is somewhat similar to https://github.com/python/cpython/issues/151497, and fits under the larger umbrella of https://github.com/python/cpython/issues/141713.
Summary
Both the pax and GNU tar families support extensions, via different mechanisms. These extensions have pre-declared lengths, and a tar parser must read a payload of length bytes to consume them.
Prior to https://github.com/python/cpython/issues/151497 this was done in a single read(n) call, resulting in a single large up-front allocation. That was changed to _safe_read with https://github.com/python/cpython/pull/151498, which bounds each read call to 1MB.
This prevents unbounded memory consumption at the read site, but not in aggregate. For example, an attacker can still contrive a pax-style tar archive with an extremely large individual pax record, and tarfile will buffer that pax record (in 1MB increments) into memory. The same is true for GNU extensions.
Solution
I think the solution is to put a reasonable caps on the sizes of extensions.
This could be done at a few different layers (e.g. restricting individual pax record sizes versus the entire pax extension size), but I think doing it at the extension size layer is probably simplest and most consistent.
My proposal would be:
- No pax or GNU extension should ever exceed 1 MB in raw size (i.e., the size reported by its tar frame). This is extremely conservative, i.e. should be well above what any real-world tar would need to put in its extensions.
- For pax in particular, the global pax extension state should never exceed some reasonable multiplier of the extension cap. For example, someone shouldn't be able to induce higher memory usage by chaining
g -> g -> g -> ... -> file.txt, where eachgmember has 1MB of pax extension state.
For prior art, we perform this kind of bounding in tar-codec, e.g. here:
and we impose a default cap of 256KB for pax extensions, 1MB for all active global pax extensions, and 128KB for GNU extensions:
(These numbers are not particularly scientific; we picked them because we think even 256KB is very conservative i.e. high for pax, and 128KB for GNU is well beyond what any normal OS will accept as a pathname length limit.)
CPython versions tested on:
CPython main branch
Operating systems tested on:
No response
貢獻指南
從這裡開始
- 先讀完整個 Issue,再讀專案的貢獻指南。
- 在 Issue 下留言說明你要接手 —— 這能避免兩個人做同樣的事。
- Fork 儲存庫,在一個分支上完成修改。
- 送出 Pull Request,並在描述裡引用這個 Issue 編號。
研究方向
從 tarfile 的 pax 和 GNU 擴充解析開始,包括現有的 _safe_read 路徑,並檢查相關測試。重現包含過大擴充或鏈結擴充的封存檔,接著為原始擴充大小和累積的全域 pax 狀態定義並測試限制,以確保解析不會讓記憶體無限制地成長。
由索引模型根據 Issue 內容生成。
評估
- 技術堆疊
- python
- 領域
- security
- Issue 類型
- 缺陷
- 難度
- 4/5
- 預估耗時
- 3-5 天
- 活躍度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 48/100