python / python/cpython

Next TAR header offset recomputation is wrong for GNU sparse 1.0 file combined with 'size' PAX header key

Open
#136,602 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stdlib type-bug
Dominant language
Python
Stars
77.2k
Forks
35.9k
PR merge metrics
PR metrics pending

Description

Bug report

Bug description:

For a more detailed description, please see #136601.

I have a bug that causes TAR file parsing to end preemptively for very large sparse files. The computed next TAR header is off by one 512 B block.

The problem is the recomputation of the next TAR offset in case the PAX header contains a size key to override the overflowed (> 8GB) TAR size:

https://github.com/python/cpython/blob/47b01da4ccedd9c00fad4325b3e87d7732abeb6d/Lib/tarfile.py#L1562-L1569

The problem is that next.offset_data is used for this recomputation even though next.offset_data gets overwritten in _proc_gnusparse_10:

https://github.com/python/cpython/blob/47b01da4ccedd9c00fad4325b3e87d7732abeb6d/Lib/tarfile.py#L1612

This leads to the next TAR offset header being off by the number of blocks it takes to store the sparse data.

But, maybe I am wrong and have overlooked something. I can say, that this fixes it for my test case:

diff --git a/Lib/tarfile.py b/Lib/tarfile.py
index 068aa13ed7..7f3e62f5a2 100644
--- a/Lib/tarfile.py
+++ b/Lib/tarfile.py
@@ -1565,7 +1565,7 @@ def _proc_pax(self, tarfile):
                 # header starts.
                 offset = next.offset_data
                 if next.isreg() or next.type not in SUPPORTED_TYPES:
-                    offset += next._block(next.size)
+                    offset += next._block(next.size) - BLOCKSIZE
                 tarfile.offset = offset
 
         return next

Minimal reproducer (tested on EXT4 with GNU tar 1.35):

echo bar > foo
echo bar > sparse
fallocate -l 9G sparse
echo bar >> sparse
fallocate --punch-hole -o 1G -l 10M sparse
tar --numeric-owner --format=pax --sparse-version=1.0 -cSf sparse.tar sparse foo
ls -la sparse.tar
# -rw-rw-r-- 1 user user 9663682560 Jul 13 14:14 sparse.tar
tar tvlf sparse.tar
# -rw-rw-r-- 1000/1000 9663676420 2025-07-13 14:13 sparse
# -rw-rw-r-- 1000/1000          4 2025-07-13 14:11 foo
python3 -c 'import sys, tarfile;
[print(tarInfo.sparse, tarInfo.offset, tarInfo.offset_data, tarInfo.size, tarInfo.name)
for tarInfo in tarfile.open(sys.argv[1])]' sparse.tar
# [(0, 1073741824), (1084227584, 8579448836), (9663676420, 0)] 0 2048 9653191172 sparse
#  -> foo is missing!
cat sparse.tar | xz -9 | zstd -19 | base64

Reproducer sparse-file-larger-than-8GiB-followed-by-normal-file.tar.xz.zst file as base64:

cat <<EOF | base64 -d | zstd -d > sparse-file-larger-than-8GiB-followed-by-normal-file.tar.xz
KLUv/QRojBIA1CP9N3pYWgAABObWtEYCACEBHAAAABDPWMz//5wCcV0AFwvGh5JaO6ePxyUOuA/z
XtE/5U/vyT1WUwqPhMr1HTeZeJyWILwrrtDwH0eKx6KKGcU7D2aYidf/9bCtFMcWp8KxDA1FLF58
w9bO4J+eDKd9QfIZFPCutpNB91dMk9bSVazx9pUcWEWn2r0SWsv1BtSYmVDmdKaMdGC/Epx8bcRA
nm5Joy2Tgi3O7VouoCAqha+1YYNOQyyB4sG+tDbfLGdW6fyZMztJ/lRFQwtlFpDLHGFpia92kkke
+2a/mwMvPc58aiT5X56QuH2mw1OhsrBKnbYYnT89BJjyAh2GTOeDbtZ/lLDGwhvxkXlnCm/M8Qiq
fUGfqAjnBeikNY2nodSBFo8YQh+636fk9xfuTQ3kKQ8qEWa613HftzHJ/X/ha1bKD91T/SPTCgd/
rhyvFtn8FBBiUS7UayidinQBNmGebczIaRsKUQKoffUTC9EbCrRXDQjQMjfDyo7N/eDIxD7jBImH
Dv8Qk/hxeFn4C83/lShGD6n8fN77mjAuVsCPhfODgcBlxCVT+PWRNjEFpbDub8FwTUcM0ZERqq1g
HbrOsScYXFmG6WZSWL7pdqxZ5OVbBQj5x9qt/PtSK3TNHlsgQvndUz34KWQJO4DLKmzftTvwxL0u
X6oPPktmQpAT+5I61gCf/xABKwDsc1On/b6ufDEan7eNMW5wnqcjX+woy4XRlZiKfiqR8id19xnA
BphNmP3Yr9WQD1EPP7IEADz8NCsncIBOR5aC/hM+FaZUAAAAAQD9/778uQYCRAAAAAEA/f85AAJE
AAAAAQD9/zkAAkQAAAABAP3/OQACRAAAAAEA/f85AAJEAAAAAQD9/zkAAkQAAAABAP3/OQACRAAA
AAEA/f85AAJEAAAAAQD9/zkAAh0GAIQLkODIaNUNHQG+Ib0xB5201x9u5Typk+S1zSY18D/tc2o+
BXKM/RM9v6MTQoFntxwNm0So6CELgft8dinBPFJBg583tJn+q69PwBnThZQjYTzvNhv0fkxX4Gjm
SgwnOrb7GU5pc2qtcrNCcHrPaNkQicmkdyzESbMAA8S2zfCiJIzpnN25EroA08/3fFWQ44Jfrake
AIiPdXLNPTRNAAGY3VWAsID7IwAAdKr/2BQXOzADAAAAAARZWgIAG0DNWVsOgERj+N4=
EOF
CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs
  • gh-136621

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read Lib/tarfile.py at _proc_pax and _proc_gnusparse_10, then run the supplied GNU tar reproducer for a GNU sparse 1.0 file with a PAX size key. Verify the next TAR header offset and confirm that the following foo entry is parsed rather than omitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.