Nullus157 / Nullus157/async-compression

Expose `set_pledged_src_size` as a public API for `zstd` compression

Open
#329 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
666
Forks
118
Avg merge
17h 18m
Merged PRs (30d)
9

Description

### Issue
`async-compression` offers no way to call [`set_pledged_src_size()`](https://github.com/gyscos/zstd-rs/blob/739b040997dca5899cafef3b3be73cd5adf4a651/src/stream/raw.rs#L355) on the `zstd` encoder. Could we provide a public API in `async-compression` to call this?

### Context

`zstd` encoding can include an optional [`Frame_Content_Size` value](https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#frame_content_size). Decompression libraries such as Python's [`zstandard`](https://python-zstandard.readthedocs.io/en/latest/) provide APIs (e.g. [`decompress`](https://python-zstandard.readthedocs.io/en/latest/decompressor.html#zstandard.ZstdDecompressor.decompress)) that use this value to decompress the data in a single operation. Without it, these libraries must treat the encoded bytes as a stream.

Writing `Frame_Content_Size` requires two steps (from the [`zstd` manual](https://facebook.github.io/zstd/zstd_manual.html)):

1. Enable `ZSTD_c_contentSizeFlag`. `async-compression` already offers [a `CParameter` method](https://docs.rs/async-compression/latest/async_compression/zstd/struct.CParameter.html#method.content_size_flag) to do this.
2. Call `ZSTD_CCtx_setPledgedSrcSize()` to set the decompressed data size. The `zstd` crate already exposes the public method [`set_pledged_src_size()`](https://github.com/gyscos/zstd-rs/blob/739b040997dca5899cafef3b3be73cd5adf4a651/src/stream/raw.rs#L355) for this, but it's not exposed by the `async-compression` crate.

When `set_pledged_src_size` is not called, the encoded bytes won't have a `Frame_Content_Size` value.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.