indygreg / indygreg/python-zstandard
python-zstandard produces incomplete data after decompress
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 642
- Forks
- 116
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 5
Description
I originally filed this against urllib3 in https://github.com/urllib3/urllib3/issues/3008 but the investigation so far is showing that urllib3 is providing the correct data to python-zstandard.
### Environment info
OS Windows-10-10.0.22621-SP0
Python 3.10.11
urllib3 2.0.1
pip freeze results:
appdirs==1.4.4
certifi==2022.9.24
charset-normalizer==2.1.1
colorama==0.4.6
flake8==5.0.4
idna==3.4
mccabe==0.7.0
numpy==1.23.4
pycodestyle==2.9.1
pydiffx==1.1
pyflakes==2.5.0
python-gitlab==3.12.0
RBTools==4.0
requests==2.28.1
requests-toolbelt==0.10.1
six==1.16.0
texttable==1.6.7
tqdm==4.64.1
typing_extensions==4.4.0
urllib3==2.0.1
zstandard==0.21.0
### Sample code
[Example source data to decompress](https://github.com/indygreg/python-zstandard/files/11376285/text.txt.zip)
I have attached a zip of text.txt.zstd because GitHub doesn't support attaching .zstd files. This file text.txt.zstd contains the raw data passed to ZstdDecoder.decompress(). Extract that text.txt.zstd file out of the zip then run:
```
import zstandard as zstd
print('Using zstandard backend: %s' % zstd.backend)
with open(r'C:\utils\temp\text.txt.zstd', 'rb') as f:
data = f.read()
print('decompressed %d into %d bytes' %
(len(data), len(zstd.decompress(data))))
```
My output is:
```
Using zstandard backend: cext
decompressed 2554 into 1048576 bytes
```
Whereas downloading zstd 1.5.5 and running `zstd.exe text.txt.zstd -d -o hi.txt` produces 8724469 bytes and a valid text file.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by extracting the attached text.txt.zstd file and running the provided Python zstandard example, then compare it with zstd 1.5.5's command-line output. Trace why zstd.decompress(data) stops at 1,048,576 bytes despite the 8,724,469-byte valid result from zstd.exe; done means the Python binding returns the complete decompressed data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100