apache / apache/pulsar

maxMessagePublishBufferSizeInMB does not consider the messages handled by Protocol Handler

Open
#12,959 4 comments 0 reactions 1 assignee Claimed by @gaozhangmin View on GitHub
lifecycle/stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Is your enhancement request related to a problem? Please describe.**
We use `maxMessagePublishBufferSizeInMB` to limit the direct memory usage for reading messages from producers, default value is 1/2 of direct memory.
If we enable any Protocol Handler in broker such as KoP, the direct memory to buffer the pending messages from Kafka producers is not limited by `maxMessagePublishBufferSizeInMB`. And if we have Pulsar producers and Kafka producers sending message together with higher throughput than BookKeeper IO, broker OOM will happen.
Although we could add another configration in KoP to limit the direct memory usage for KoP, we could not use the entire memory space since we will have two separate configration for pulsar and KoP.

**Describe the solution you'd like**
To make effective use of memory, using a LongAdder in `BrokerService` to record the current pending bytes for entire broker, include both pulsar pending messages and Protocol Handler's pending messages.
When handling producer message, we could check if pending bytes over `maxMessagePublishBufferSizeInMB` and stop reading from producer both in broker and Protocol Handler side.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.