zstd::bulk::decompress(.., capacity) immediately allocates a Vec that size
- Dominant language
- Rust
- Stars
- 655
- Forks
- 136
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 4
Description
Hey there,
I misinterpreted this parameter to mean a maximum amount of data to decompress, instead it is directly controlling a Vec allocation, resulting in a surprising OOM much later, and possibly also a hidden perf issue as system calls and (kernel mmap_sem etc locking) will likely be involved for any large value.
The unused Vec capacity is also not trimmed. It's possible (though I guess unlikely) with a user running this in a loop for many small strings but a large capacity to discover either slowness or surprising breakage.
Would suggest perhaps documenting the dual meaning of capacity. I'm not sure what a more desirable behaviour would be - arguably the current behaviour is fine, particularly for cases where the only alternative would be incremental reallocation and memmoves killing some of the perf benefit to using zstd at all.
Thanks for a great library
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the public zstd::bulk::decompress(.., capacity) API and its existing documentation. Clarify that capacity controls the initial Vec allocation and is not only a decompression maximum; done when the documented behavior reflects this distinction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100