indygreg / indygreg/python-zstandard
How to compress (or decompress) a large file (100G or more) into one file or multiple files using zstd?
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 642
- Forks
- 116
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 5
Description
I read readme, I know that simple-api is prone to abuse, as The input and output values must fit in memory, so passing in very large values can result in excessive memory usage. For this reason, one of the streaming based APIs is preferred for larger values.
But I didn't understand how to use the streaming API to compress a large file into one or more files.
And what should I do when decompressing?
I hope someone can provide a simple example(use python) to give me a reference. Thanks.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the README sections covering the simple and streaming APIs, then identify the streaming examples or entry points for file input and output. The documentation should provide a simple Python example for compressing and decompressing very large files, including how to produce one or multiple output files without loading the full data into memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100