indygreg / indygreg/python-zstandard
Does compressor update dictionary?
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 642
- Forks
- 116
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 5
Description
Sorry for using issues for questions, but the generic forum has no ideas.
Does ZstdCompressor with dictionary add new data to dictionary when it runs compress() or is dictionary only limited to data that was used for its creation? In other words, if I build a dictionary from A, and then use it to compress B and C with .compress(), will it help the compressor to find similarities between B and C ?
Use case: I receive a big set of small (4kb-16kb) pieces of data from a remote device. They may or may not share a lot of similar sequences between themselves. I look for a way to compress them on the fly, without first storing them raw. I need to be able to decompress any piece by itself, so streaming API can't help me because of all this tails and flush() stuff.
Thanks for the help!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ZstdCompressor, compress(), dictionary, streaming API, and flush() behavior named in the issue. Done means establishing whether compression updates the dictionary across independent pieces and documenting the answer for this use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100