gyscos / gyscos/zstd-rs

zstd::bulk::decompress(.., capacity) immediately allocates a Vec that size

Open
#331 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
655
Forks
136
Avg merge
2d 16h
Merged PRs (30d)
4

Description

Hey there,

I misinterpreted this parameter to mean a maximum amount of data to decompress, instead it is directly controlling a Vec allocation, resulting in a surprising OOM much later, and possibly also a hidden perf issue as system calls and (kernel mmap_sem etc locking) will likely be involved for any large value.

The unused Vec capacity is also not trimmed. It's possible (though I guess unlikely) with a user running this in a loop for many small strings but a large capacity to discover either slowness or surprising breakage.

Would suggest perhaps documenting the dual meaning of capacity. I'm not sure what a more desirable behaviour would be - arguably the current behaviour is fine, particularly for cases where the only alternative would be incremental reallocation and memmoves killing some of the perf benefit to using zstd at all.

Thanks for a great library

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the public zstd::bulk::decompress(.., capacity) API and its existing documentation. Clarify that capacity controls the initial Vec allocation and is not only a decompression maximum; done when the documented behavior reflects this distinction.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
documentation
Issue type
Documentation
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.