indygreg / indygreg/python-zstandard

Does compressor update dictionary?

Open
#104 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
C
Stars
642
Forks
116
Avg merge
1d 14h
Merged PRs (30d)
5

Description

Sorry for using issues for questions, but the generic forum has no ideas.

Does ZstdCompressor with dictionary add new data to dictionary when it runs compress() or is dictionary only limited to data that was used for its creation? In other words, if I build a dictionary from A, and then use it to compress B and C with .compress(), will it help the compressor to find similarities between B and C ?

Use case: I receive a big set of small (4kb-16kb) pieces of data from a remote device. They may or may not share a lot of similar sequences between themselves. I look for a way to compress them on the fly, without first storing them raw. I need to be able to decompress any piece by itself, so streaming API can't help me because of all this tails and flush() stuff.

Thanks for the help!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ZstdCompressor, compress(), dictionary, streaming API, and flush() behavior named in the issue. Done means establishing whether compression updates the dictionary across independent pieces and documenting the answer for this use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.