KillingSpark / KillingSpark/zstd-rs

Support resumable decompression

Open
#41 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
449
Forks
66
PR merge metrics
No merged PRs in 30d

Description

I want to be able to decode all or part of a file and then rewind (Seek) to some earlier point in the file and re-decode it. This would enable the file to be read as random-access, once the initial read was done.

Doing this requires being able to save the state of some earlier decode point. I think I can clone the FrameDecoder (once I make it and its children Cloneable) and use this and the source file position to restore my read to some earlier point.

Supporting this mode will make it possible to support something almost like [zstd-seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md) without first having to compress the file in a multi-frame format.

libz has [support for this](https://github.com/madler/zlib/blob/master/examples/zran.c); it's seldom used but it comes in handy when needed.

I can work on this some myself but I could use some guidance. For example, I suspect I don't need to clone the entire `FrameDecoder` struct to make this work; I don't think I need all the buffers it holds. I may only need the relevant context information (dictionaries, bit positions, etc.) and some feeder data.

I wouldn't mind only saving state at the end of a block, but I'm not sure if that's advantageous.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the FrameDecoder entry point and the linked zlib zran.c and zstd-seekable format references. Determine what decoder state and source position must be retained for rewind and re-decode, then verify that a previously read file can be resumed from an earlier point.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.