python / python/cpython

tarfile: unbounded memory use on large pax and GNU extensions

Aperta
#155,633 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

stdlib type-bug
Lingua principale
Python
Stelle
77.2k
Fork
35.9k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

Bug report

Bug description:

This is somewhat similar to https://github.com/python/cpython/issues/151497, and fits under the larger umbrella of https://github.com/python/cpython/issues/141713.

Summary

Both the pax and GNU tar families support extensions, via different mechanisms. These extensions have pre-declared lengths, and a tar parser must read a payload of length bytes to consume them.

Prior to https://github.com/python/cpython/issues/151497 this was done in a single read(n) call, resulting in a single large up-front allocation. That was changed to _safe_read with https://github.com/python/cpython/pull/151498, which bounds each read call to 1MB.

This prevents unbounded memory consumption at the read site, but not in aggregate. For example, an attacker can still contrive a pax-style tar archive with an extremely large individual pax record, and tarfile will buffer that pax record (in 1MB increments) into memory. The same is true for GNU extensions.

Solution

I think the solution is to put a reasonable caps on the sizes of extensions.

This could be done at a few different layers (e.g. restricting individual pax record sizes versus the entire pax extension size), but I think doing it at the extension size layer is probably simplest and most consistent.

My proposal would be:

  1. No pax or GNU extension should ever exceed 1 MB in raw size (i.e., the size reported by its tar frame). This is extremely conservative, i.e. should be well above what any real-world tar would need to put in its extensions.
  2. For pax in particular, the global pax extension state should never exceed some reasonable multiplier of the extension cap. For example, someone shouldn't be able to induce higher memory usage by chaining g -> g -> g -> ... -> file.txt, where each g member has 1MB of pax extension state.

For prior art, we perform this kind of bounding in tar-codec, e.g. here:

https://github.com/astral-sh/tar-codec/blob/dbd4b5efeb6edb732c993d107a7cfdc83f8d29a8/crates/tar-framing/src/stream.rs#L1099-L1154

and we impose a default cap of 256KB for pax extensions, 1MB for all active global pax extensions, and 128KB for GNU extensions:

https://github.com/astral-sh/tar-codec/blob/dbd4b5efeb6edb732c993d107a7cfdc83f8d29a8/crates/tar-framing/src/lib.rs#L124-L137

(These numbers are not particularly scientific; we picked them because we think even 256KB is very conservative i.e. high for pax, and 128KB for GNU is well beyond what any normal OS will accept as a pathname length limit.)

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia dall’analisi delle estensioni pax e GNU di tarfile, incluso il percorso _safe_read esistente, e esamina i test correlati. Riproduci archivi con estensioni sovradimensionate o concatenate, quindi definisci e testa dei limiti per la dimensione delle estensioni grezze e per lo stato pax globale accumulato, in modo che l’analisi non possa far crescere la memoria senza limiti.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
security
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.