apache / apache/pulsar-client-go

The aggregation of the CompressionProvider to prevent the large numbers of memory used by the underlying Zstd Libs

Open
#1,344 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

**Is your feature request related to a problem? Please describe.**

Backgroud: In the scene of log collecting, a go application will collecting logs from many different business sides, and produce to individual topics in Pulsar. So a go application have to create many topic producers, and each may have many partition producers.

Problem: The implementation of the PartitionProducer each holds an CompressionProvider, which eventually holds an instance of the zstdprovider(for pure go context, this may use the github.com/klauspost/compress/zstd).
The zstdprovider will each occupy a large byte slice for storing the history data to improve the compress ratio. And this will only release when the partition producer close. But the partition producer will never close except for the topic producer to be closed.
So, this will lead to the slow(or quick) increasing of the memory of the Go application, which may eventually lead to the frequent OOM of the Go application. It just seems like the leaking of the memory in the partition strategy of RoundRobin( and other partition strategies, which may change the producing partition with the time).

**Describe the solution you'd like**
1# Just reduce the Compression Provider according to the CompressionLevel will solve this problem.

Contributor guide

Open the contributing guide

Research direction

Trace the PartitionProducer, CompressionProvider, and zstdprovider relationship, focusing on how RoundRobin partition changes retain compression state and how CompressionLevel affects it. Confirm the memory-retention behavior and define completion as reducing the reported per-partition memory growth without changing producer behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
performance, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.