HDFGroup / HDFGroup/hdf5_plugins
Update Zstd decompression for "unknown decompressed size" when streaming API was used for compression
- Dominant language
- CMake
- Stars
- 21
- Forks
- 26
- Avg merge
- 5d 7h
- Merged PRs (30d)
- 1
Description
# Introduction
The Zstandard plugin for HDF5 should be modified to allow for an unknown decompressed size in the frame header.
Currently, the Zstd decompression scheme, following from the [original implemention](https://github.com/aparamon/HDF5Plugin-Zstandard), uses `ZSTD_getDecompressedSize` to obtain the size of the decompressed buffer. The returned value is not validated and passed directly to `malloc`.
https://github.com/HDFGroup/hdf5_plugins/blob/770d70ae73587714629cf5ec139d482c1562e7c1/ZSTD/src/H5Zzstd.c#L59-L60
`ZSTD_getDecompressedSize` returns `0` if the decompressed size is empty, unknown, or an error has occured. If `malloc` is asked to allocate `0` bytes, it will return `NULL`, resulting in returning an error condition. This is an incorrect result if the decompressed size is actually empty or unknown and there is no actual error. `ZSTD_getDecompressedSize` is obsolete.
`ZSTD_getFrameContentSize` should replace the use of `ZSTD_getDecompressedSize`. `ZSTD_getFrameContentSize` distinguishes between empty, unknown, or an error. The unknown or error states are indicated by a return value of `ZSTD_CONTENTSIZE_UNKNOWN` or `ZSTD_CONTENTSIZE_ERROR`, respectively.
The unknown decompression state is common. This occurs when the compression is done via the streaming API via `ZSTD_compressStream` or `ZSTD_compressStream2`. `ZSTD_compressStream2` in particular only stores the frame size when either `ZSTD_e_end` is provided on the initial call or `ZSTD_CCtx_setPledgedSrcSize` is used.
# Tasks
- [x] Use `ZSTD_getFrameContentSize` instead of the obsolete `ZSTD_getDecompressedSize` to correctly distinguish between empty, unknown, or error states when determining the decompressed size.
- [x] Recognize the unknown size state by checking the return value of `ZSTD_getFrameContentSize` against `ZSTD_CONTENTSIZE_UNKNOWN`
- [ ] Address unknown size state by growing buffer if needed or using stream decompression API via `ZSTD_decompressStream`
- The HDF5 library should know the expected number of bytes for a chunk. We should not be relying on the filter to figure this out.
# References
[1] https://facebook.github.io/zstd/zstd_manual.html
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.