indygreg / indygreg/python-zstandard

decompressobj inefficiency and work-around, patch included.

Open
#29 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
642
Forks
116
Avg merge
1d 14h
Merged PRs (30d)
5

Description

Greetings!

I'm working with a network protocol which is essentially a zstandard-compressed stream of newline-delimited lines of text. A Twisted protocol for receiving this data is given a slice of compressed data which can span frames. It's simple to feed chunks of data to `decompressobj.decompress` until it yields uncompressed data, but it's not clear how to determine how much trailing input data didn't contribute to the uncompressed output and should be processed with a new instance of `decompressobj`. The only solution I've found in terms of python-zstandard 0.8.1 has been to feed data to `decompressobj.decompress` one byte at a time and roll to a new `decompressobj` each time uncompressed data is produced. This is pretty slow. If I've missed something, I'd appreciate advice.

Meanwhile, I've privately replaced `decompressobj.decompress` with a new function `decompressobj.decompress2` and re-implemented `decompressobj.decompress` as a trivial wrapper around this new function. The new function returns a tuple consisting of (1) the uncompressed `result` and (2) the value of `input.pos` before the final call to `ZSTD_decompressStream`. Using this interface, my protocol's `dataReceived` method looks like this:

```
def dataReceived(self, bytes):
decompressed, remaining = self.__dobj.decompress2(bytes)
if len(decompressed) > 0:
self.__json_receiver.dataReceived(decompressed)
self.__dobj = self.__dctx.decompressobj()
if remaining > 0:
self.dataReceived(bytes[remaining:])
```

If I haven't missed something in the current python-zstandard API, would you consider a solution like mine for inclusion in the next release? I'm attaching the delta to `decompressobj.c` for reference.

Thanks!

[decompressobj.c.diff.txt](https://github.com/indygreg/python-zstandard/files/1245709/decompressobj.c.diff.txt)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review decompressobj.c and the attached decompressobj.c.diff.txt, beginning with the existing decompressobj.decompress implementation and its ZSTD_decompressStream calls. Reproduce the reported stream case with chunks spanning frames, then confirm the proposed input-position behavior preserves the existing API and lets callers identify remaining input.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python
Domain
backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.