elastic / elastic/logstash

pq: improve zstd by using shared contexts

Open
#18,210 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

In #18121 we add support for ZSTD compression using the convenience methods `com.github.luben.zstd.Zstd#decompress(byte[])` convenience method that does not require a-priori knowledge about the contents of the received `byte[]` or the expected decompressed size of the returned `byte[]`, but carry the cost of instantiating a new `com.github.luben.zstd.ZstdDecompressCtx` for each usage instead of allowing the reuse of pooled resources.

This convenience method relies on _private_ methods within `com.github.luben.zstd.Zstd` to scan the frames of the provided `byte[]` to determine the result buffer size, so that they can invoke methods on the reusable `com.github.luben.zstd.ZstdDecompressCtx` that require that foreknowledge.

This task is to explore the benefit of reusing `ZstdDecompressCtx`, which will entail:
- reimplementing those private methods from the BSD-2-licensed upstream project
- implementing a resource pool, using `ThreadPool`+`Cleaner` or similar
- analyzing the delta in performance and/or resource usage

Contributor guide

Open the contributing guide

Research direction

Start by reviewing issue #18121 and the com.github.luben.zstd.Zstd convenience methods described here, then compare creating a ZstdDecompressCtx per use with reusing pooled contexts. The work is complete when the frame-scanning methods are reimplemented, a resource-pooling approach is in place, and the performance and resource-usage delta has been analyzed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.