indygreg / indygreg/python-zstandard

python-zstandard produces incomplete data after decompress

Open
#196 19 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C
Stars
642
Forks
116
Avg merge
1d 14h
Merged PRs (30d)
5

Description

I originally filed this against urllib3 in https://github.com/urllib3/urllib3/issues/3008 but the investigation so far is showing that urllib3 is providing the correct data to python-zstandard.

### Environment info

OS Windows-10-10.0.22621-SP0
Python 3.10.11
urllib3 2.0.1

pip freeze results:

appdirs==1.4.4
certifi==2022.9.24
charset-normalizer==2.1.1
colorama==0.4.6
flake8==5.0.4
idna==3.4
mccabe==0.7.0
numpy==1.23.4
pycodestyle==2.9.1
pydiffx==1.1
pyflakes==2.5.0
python-gitlab==3.12.0
RBTools==4.0
requests==2.28.1
requests-toolbelt==0.10.1
six==1.16.0
texttable==1.6.7
tqdm==4.64.1
typing_extensions==4.4.0
urllib3==2.0.1
zstandard==0.21.0

### Sample code

[Example source data to decompress](https://github.com/indygreg/python-zstandard/files/11376285/text.txt.zip)

I have attached a zip of text.txt.zstd because GitHub doesn't support attaching .zstd files. This file text.txt.zstd contains the raw data passed to ZstdDecoder.decompress(). Extract that text.txt.zstd file out of the zip then run:

```
import zstandard as zstd

print('Using zstandard backend: %s' % zstd.backend)

with open(r'C:\utils\temp\text.txt.zstd', 'rb') as f:
data = f.read()

print('decompressed %d into %d bytes' %
(len(data), len(zstd.decompress(data))))
```
My output is:
```
Using zstandard backend: cext
decompressed 2554 into 1048576 bytes
```
Whereas downloading zstd 1.5.5 and running `zstd.exe text.txt.zstd -d -o hi.txt` produces 8724469 bytes and a valid text file.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by extracting the attached text.txt.zstd file and running the provided Python zstandard example, then compare it with zstd 1.5.5's command-line output. Trace why zstd.decompress(data) stops at 1,048,576 bytes despite the 8,724,469-byte valid result from zstd.exe; done means the Python binding returns the complete decompressed data.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.