apache / apache/pulsar

the bookie heap used not balance after run a while and cause one bookie OOM.

Open
#16,387 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
We have four bookie node, all the parameter setting was the same, but the bookie heap used not balance after run a while, one bookie head memory used high and the gc time enlarge and frequencily , this can risk bookie OOM.

**version**: pulsar-all:2.9.1

PULSAR_GC: |
-XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=4 -XX:ConcGCThreads=4 -XX:G1NewSizePercent=30 -XX:G1MaxNewSizePercent=60 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -verbosegc
PULSAR_MEM: |
-Xms10g -Xmx10g -XX:MaxDirectMemorySize=7g

besides,we have 700,00 topics, and 40000 connections.

**To Reproduce**
out request is stable, pulsar run about two weeks the blance issue comes. and the issue comes again after two weeks when restart the bookie node.

**Expected behavior**
the bookie heap used balanced in all node.

**Screenshots**
![image](https://user-images.githubusercontent.com/5387559/177235971-29aae1c4-7756-4c4c-99e9-de20824a7df9.png)

**Desktop (please complete the following information):**
- OS: [e.g. iOS]
Ubuntu 20.04.4 LTS ( kubernetes pod)

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

The report names no source files, tests, or entry points. Start by reviewing the two-week reproduction details, Bookie heap and GC settings, topic and connection counts, and the attached screenshot; done means explaining the imbalance and preventing the affected bookie from reaching OOM.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes, ubuntu
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.