indygreg / indygreg/python-zstandard
decompressobj inefficiency and work-around, patch included.
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 642
- Forks
- 116
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 5
Description
Greetings!
I'm working with a network protocol which is essentially a zstandard-compressed stream of newline-delimited lines of text. A Twisted protocol for receiving this data is given a slice of compressed data which can span frames. It's simple to feed chunks of data to `decompressobj.decompress` until it yields uncompressed data, but it's not clear how to determine how much trailing input data didn't contribute to the uncompressed output and should be processed with a new instance of `decompressobj`. The only solution I've found in terms of python-zstandard 0.8.1 has been to feed data to `decompressobj.decompress` one byte at a time and roll to a new `decompressobj` each time uncompressed data is produced. This is pretty slow. If I've missed something, I'd appreciate advice.
Meanwhile, I've privately replaced `decompressobj.decompress` with a new function `decompressobj.decompress2` and re-implemented `decompressobj.decompress` as a trivial wrapper around this new function. The new function returns a tuple consisting of (1) the uncompressed `result` and (2) the value of `input.pos` before the final call to `ZSTD_decompressStream`. Using this interface, my protocol's `dataReceived` method looks like this:
```
def dataReceived(self, bytes):
decompressed, remaining = self.__dobj.decompress2(bytes)
if len(decompressed) > 0:
self.__json_receiver.dataReceived(decompressed)
self.__dobj = self.__dctx.decompressobj()
if remaining > 0:
self.dataReceived(bytes[remaining:])
```
If I haven't missed something in the current python-zstandard API, would you consider a solution like mine for inclusion in the next release? I'm attaching the delta to `decompressobj.c` for reference.
Thanks!
[decompressobj.c.diff.txt](https://github.com/indygreg/python-zstandard/files/1245709/decompressobj.c.diff.txt)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review decompressobj.c and the attached decompressobj.c.diff.txt, beginning with the existing decompressobj.decompress implementation and its ZSTD_decompressStream calls. Reproduce the reported stream case with chunks spanning frames, then confirm the proposed input-position behavior preserves the existing API and lets callers identify remaining input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100