HDFGroup / HDFGroup/hdf5_plugins

Update Zstd decompression for "unknown decompressed size" when streaming API was used for compression

Open
#116 17 comments 0 reactions 1 assignee Assigned to @ajelenak View on GitHub
Filter - ZSTD Priority - 1. High 🔼 Type - Improvement
Dominant language
CMake
Stars
21
Forks
26
Avg merge
5d 7h
Merged PRs (30d)
1

Description

# Introduction

The Zstandard plugin for HDF5 should be modified to allow for an unknown decompressed size in the frame header.

Currently, the Zstd decompression scheme, following from the [original implemention](https://github.com/aparamon/HDF5Plugin-Zstandard), uses `ZSTD_getDecompressedSize` to obtain the size of the decompressed buffer. The returned value is not validated and passed directly to `malloc`.

https://github.com/HDFGroup/hdf5_plugins/blob/770d70ae73587714629cf5ec139d482c1562e7c1/ZSTD/src/H5Zzstd.c#L59-L60

`ZSTD_getDecompressedSize` returns `0` if the decompressed size is empty, unknown, or an error has occured. If `malloc` is asked to allocate `0` bytes, it will return `NULL`, resulting in returning an error condition. This is an incorrect result if the decompressed size is actually empty or unknown and there is no actual error. `ZSTD_getDecompressedSize` is obsolete.

`ZSTD_getFrameContentSize` should replace the use of `ZSTD_getDecompressedSize`. `ZSTD_getFrameContentSize` distinguishes between empty, unknown, or an error. The unknown or error states are indicated by a return value of `ZSTD_CONTENTSIZE_UNKNOWN` or `ZSTD_CONTENTSIZE_ERROR`, respectively.

The unknown decompression state is common. This occurs when the compression is done via the streaming API via `ZSTD_compressStream` or `ZSTD_compressStream2`. `ZSTD_compressStream2` in particular only stores the frame size when either `ZSTD_e_end` is provided on the initial call or `ZSTD_CCtx_setPledgedSrcSize` is used.

# Tasks

- [x] Use `ZSTD_getFrameContentSize` instead of the obsolete `ZSTD_getDecompressedSize` to correctly distinguish between empty, unknown, or error states when determining the decompressed size.
- [x] Recognize the unknown size state by checking the return value of `ZSTD_getFrameContentSize` against `ZSTD_CONTENTSIZE_UNKNOWN`
- [ ] Address unknown size state by growing buffer if needed or using stream decompression API via `ZSTD_decompressStream`
- The HDF5 library should know the expected number of bytes for a chunk. We should not be relying on the filter to figure this out.

# References

[1] https://facebook.github.io/zstd/zstd_manual.html

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.