indygreg / indygreg/python-zstandard
Feature request: allow `decompress` to decompress stream data without content size in the header
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 642
- Forks
- 116
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 5
Description
Edit: this is documented, but I think it would be an useful addition. Maybe by passing max_output_size=-1?
The example below raises ZstdError: could not determine content size in frame header
import io
import zstandard as zstd
data = b'some data'
input_buffer = io.BytesIO(data)
with zstd.ZstdCompressor().stream_reader(input_buffer) as r:
compressed = r.read()
decompressed = zstd.decompress(compressed)
assert decompressed == data
Passing max_output_size to the decompress call works as expected:
...
decompressed = zstd.decompress(compressed, max_output_size=len(data))
assert decompressed == data
Using ZstdDecompressor.stream_reader also works:
...
with zstd.ZstdDecompressor().stream_reader(io.BytesIO(compressed)) as r:
decompressed = r.read()
assert decompressed == data
I think decompress should handle this case as well, since it's not always possible to known how the data was compressed.
The package version I use is zstandard 0.15.2 on windows 10 (but I don't think it's os dependent)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the examples with zstd.decompress, max_output_size, and ZstdDecompressor.stream_reader to compare current behavior. Start at the decompress API entry point and its tests; done means decompressing frames without a content size succeeds without requiring a known output size, with coverage for the reported case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100