apache / apache/pulsar

[BUG] Questions about pulsar broker direct OOM

Open
#12,169 6 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
A clear and concise description of what the bug is.

Pulsar and bookkeeper version:
pulsar-2.8.0 and pulsar-2.8.0 built-in bookkeeper
cluster with 5 brokers and 5 bookies

In order to figure out the reason for the OOM of the pulsar broker's direct memory, I tested different scenarios and got some different results.

After analyzing the pulsar broker heap dump, a large number of PendingAddOp instances have not been recycled or destroyed.

As shown in the figure below, I suspect that a large number of entry requests written to bookie have not received all the WQ responses, which makes PendingAddOp unable to be recycled or destroyed.

![image](https://user-images.githubusercontent.com/35599757/134630405-64fec4d3-2714-4c8a-968b-bf65e6bd75bd.png)

Therefore, I use maxMessagePublishBufferSizeInMB to limit the traffic handled by the broker according to https://github.com/apache/pulsar/pull/7406 and https://github.com/apache/pulsar/pull/6178.

But next is my test results:
1. The broker is configured with maxMessagePublishBufferSizeInMB=512, EW A=3:3:2, OOM still occurs after the pressure test
2. The broker configures maxMessagePublishBufferSizeInMB=512, and tests EW A=3:3:3, 3:2:2, and 2:2:2 respectively. After the pressure test, the direct memory is normal
3. The broker configures maxMessagePublishBufferSizeInMB=2048, test EW A=3:3:3 and 3:2:2, after the pressure test, the direct memory is normal
4. The broker configuration keeps maxMessagePublishBufferSizeInMB as the default value, the default is 1/2 of the maximum allocated off-heap memory (8/2=4GB in the test), test EW A=3:3:3 and 3:2:2, pressure test The off-heap memory is normal
5. The broker configures maxMessagePublishBufferSizeInMB=-1, closes current limiting measures, tests EW A=3:3:3 and 3:2:2, the memory is normal after the pressure test
6. The broker configures maxMessagePublishBufferSizeInMB=-1, closes current limiting measures, tests EW A=3:3:2, OOM occurs after the pressure test

The next questions also are related to #9562

My question is, whether maxMessagePublishBufferSizeInMB is configured or not,
as long as AQ=WQ, direct memory is normal,
as long as AQ

Contributor guide

Open the contributing guide

Research direction

Start with the broker's PendingAddOp handling and the bookie write-response path described in the issue, then review maxMessagePublishBufferSizeInMB and the linked workload YAML. Reproduce the AQ=WQ and AQ

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.