pypa / pypa/setuptools

archive_util bypasses tarfile's PEP 706 extraction filters by calling private APIs

Open
#5,328 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.9k
Forks
1.4k
Avg merge
1d 1h
Merged PRs (30d)
1

Description

setuptools.archive_util.unpack_tarfile extracts through three private tarfile APIs, which means none of the hardening CPython added in PEP 706 ever runs.

# unpack_tarfile
tarobj._extract_member(member, final_dst)

# _iter_open_tar
tar_obj.chown = lambda *args: None

# _resolve_tar_file_or_dir
tar_member_obj = tar_obj._getmember(linkpath)
Why it matters

_extract_member is the internal worker that extractall calls after the extraction filter has vetted and rewritten the member. Calling it directly skips the filter entirely, so setuptools gets none of data_filter's protections: absolute and upward link targets, device and FIFO members, setuid/setgid/sticky bits, and permission handling. tar_obj.chown = lambda *args: None is a hand-rolled substitute for one small piece of what the filter already does properly.

The containment check added in #5325 re-implements one of those protections by hand. That was the right scope for a security fix, but it means we now maintain our own partial copy of logic the stdlib maintains fully, and the remaining gaps stay open.

Being private API is a liability in its own right: signature or behavior can change in any release with no deprecation, and _getmember in particular has no stability guarantee.

The complication

unpack_tarfile cannot simply become tarobj.extractall(extract_dir, filter='data'). The documented progress_filter callback takes (source_path, dest_path) and may return a different destination or None to skip, so callers can redirect individual members anywhere they like — including, legitimately, outside the nominal extraction directory. That is why #5325 applies its containment check to the preliminary destination, before progress_filter runs.

Reconciling the two is the actual work here. Rough options:

  • Run each member through tarfile.data_filter(member, extract_dir) for validation and to pick up its sanitized TarInfo, then extract to the possibly-redirected destination. Keeps progress_filter semantics; still uses _extract_member for the write.
  • Extract to a temporary directory with extractall(filter='data'), then move members into place according to progress_filter. Fully public API, at the cost of an extra copy and of changing when the callback is invoked relative to extraction.
  • Reduce the progress_filter contract to filtering only, not redirection — a breaking change, but it would let the stdlib own extraction entirely.

tarfile.data_filter is available on all currently supported Pythons (requires-python = ">=3.10") via the 3.10.12 / 3.11.4 security backports, though the exact minimum patch level is worth confirming before relying on it unconditionally.

Split out of #5325, where this was noted but deliberately left alone as out of scope for the security fix.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with setuptools.archive_util.unpack_tarfile, _iter_open_tar, and _resolve_tar_file_or_dir, then inspect tarfile.data_filter and the changes from #5325. Confirm the minimum supported Python patch levels before choosing an approach. Done means avoiding the private tarfile APIs while preserving progress_filter's per-member redirection and skip behavior, with coverage for the security protections described here.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security, tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.