KillingSpark / KillingSpark/zstd-rs
Support resumable decompression
- Dominant language
- Rust
- Stars
- 449
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Description
I want to be able to decode all or part of a file and then rewind (Seek) to some earlier point in the file and re-decode it. This would enable the file to be read as random-access, once the initial read was done.
Doing this requires being able to save the state of some earlier decode point. I think I can clone the FrameDecoder (once I make it and its children Cloneable) and use this and the source file position to restore my read to some earlier point.
Supporting this mode will make it possible to support something almost like [zstd-seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md) without first having to compress the file in a multi-frame format.
libz has [support for this](https://github.com/madler/zlib/blob/master/examples/zran.c); it's seldom used but it comes in handy when needed.
I can work on this some myself but I could use some guidance. For example, I suspect I don't need to clone the entire `FrameDecoder` struct to make this work; I don't think I need all the buffers it holds. I may only need the relevant context information (dictionaries, bit positions, etc.) and some feeder data.
I wouldn't mind only saving state at the end of a block, but I'm not sure if that's advantageous.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the FrameDecoder entry point and the linked zlib zran.c and zstd-seekable format references. Determine what decoder state and source position must be retained for rewind and re-decode, then verify that a previously read file can be resumed from an earlier point.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100