Size tresholds for changing the compression parameters penalizes 256K inputs
- Dominant language
- C
- Stars
- 27.9k
- Forks
- 2.6k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 8
Description
**Description**
The problem I am having is related to raster compression in GIS and web maps and machine learning, where it is common to keep raster data in individually compressed tiles, usually 256x256 or 512x512 pixels. An RGBA byte tile at 256x256 pixels, or single byte 512x512 tile is exactly 256K. Floating point or integer tiles at 256x256 are also 256K.
When using the simple API, ZSTD compression parameters are chosen based on the level parameter and the input size. There are four sets of such parameters, for input size > 256K, > 128K, > 16K and <= 16K.
Based on those thresholds, the 256K size falls at the edge of the second tier. Experimentally I found out that if I make the input tiles slightly larger than 256K, thus switching to the first tier, the middle level compression is twice as fast while the result is smaller by a few percent. This is surprising but expected, based on the description of the match finder introduced in version 1.5.
The size limits where the parameter set changes seem arbitrary, and in this particular but common case the current values hurts compression performance considerably. Since power of 2 buffer sizes are very common, they should be optimized for in terms of compression and speed, which doesn't seem to be the case here. Possibly it was not as large of a performance difference in version 1.5.0, but it is definitely noticeable now. It likely depends on the input data too, but in all the cases I've tested using the first tier parameters is better than using the second tier.
**Proposed solution**
The immediate solution is to move the 256K into the first tier. I found that this can be done by changing a single line in lib/compress/zstd_compress.c from
`U32 const tableID = (rSize <= 256 KB) + (rSize <= 128 KB) + (rSize <= 16 KB);`
to
`U32 const tableID = (rSize < 256 KB) + (rSize <= 128 KB) + (rSize <= 16 KB);`
As far as I can tell this has the desired effect and doesn't affect decoding using existing binaries.
This kind of change might be applicable to the 128K and 16K. It might be useful to lower the 256K threshold to 192K, to include the even more common RGB 256x256 tile.
**Alternatives**
A slightly more flexible approach would be to allow the tier limits to be chosen at compile time. Since most users will not change the defaults, this should be done in addition to making the 256K input size fall in first tier.
Using the streaming API seems to avoid this problem entirely and using the first tier parameters. I'm sure that using the advanced API can handle this issue. Yet using the simple API should not result in such a large drop in performance.
Contributor guide
Assessment
This issue has not been assessed yet.