apache / apache/pulsar

Bookie Journal Queue Length is increasing gradually during performance test executions

Open
#6,479 2 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
Bookie Journal Queue Length is increasing gradually during performance executions
**To Reproduce**
Steps to reproduce the behavior:
1. Install Pulsar 2.5.0-08 on kubernetes cluster environment ( 3 Bookies, 3 brokers,3 proxy).
2. Create a topic with three partitions.
2. Run longevity test (48 hours) by passing 1 KB message size as input to send data to created topic in step 2
3. After completion of the few hours execution, Check size of "Journal Queue Length" bookie metric.

**Expected behavior**
Should not increase queue length of journal and bookies should be able to handle properly.

**Screenshots**

Attached screenshot of "Journal Queue Size Length" metric.

**Desktop (please complete the following information):**
- OS: CentOS 7.7
- Master Node: 1
- Worker Nodes : 3

**H/w Details**:
- Master node: 12x16x200 (Not running any Pulsar pods on master node)
- Worker nodes : 16x32x200 ( each worker node has additional 4 virtual disks for Pulsar pods)

Contributor guide

Open the contributing guide

Research direction

No source files or tests are named. Start by reproducing the Pulsar 2.5.0-08 Kubernetes setup with three bookies, three brokers, and three proxies, then run the 48-hour longevity test while monitoring the Bookie Journal Queue Length metric. Done means the journal queue does not increase gradually and the bookies continue handling the workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
distributed-systems, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.