pq: improve zstd by using shared contexts
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
In #18121 we add support for ZSTD compression using the convenience methods `com.github.luben.zstd.Zstd#decompress(byte[])` convenience method that does not require a-priori knowledge about the contents of the received `byte[]` or the expected decompressed size of the returned `byte[]`, but carry the cost of instantiating a new `com.github.luben.zstd.ZstdDecompressCtx` for each usage instead of allowing the reuse of pooled resources.
This convenience method relies on _private_ methods within `com.github.luben.zstd.Zstd` to scan the frames of the provided `byte[]` to determine the result buffer size, so that they can invoke methods on the reusable `com.github.luben.zstd.ZstdDecompressCtx` that require that foreknowledge.
This task is to explore the benefit of reusing `ZstdDecompressCtx`, which will entail:
- reimplementing those private methods from the BSD-2-licensed upstream project
- implementing a resource pool, using `ThreadPool`+`Cleaner` or similar
- analyzing the delta in performance and/or resource usage
Contributor guide
Research direction
Start by reviewing issue #18121 and the com.github.luben.zstd.Zstd convenience methods described here, then compare creating a ZstdDecompressCtx per use with reusing pooled contexts. The work is complete when the frame-scanning methods are reimplemented, a resource-pooling approach is in place, and the performance and resource-usage delta has been analyzed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100