apache / apache/pulsar

[Bug] When batching is enabled even when no data is being streamed CPU usage remains high post streaming to 1.5k topics

Open
#23,574 2 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

2.10.6

### Minimal reproduce step

With batching enabled stream data to 1500 topics, then stop streaming and monitor the CPU usage
When batching is enabled
![image](https://github.com/user-attachments/assets/33cf9774-d4d2-49c8-b436-7d30eaf83154)

![image](https://github.com/user-attachments/assets/9934c434-69d5-4273-8a61-751d654660e7)

When batching is disabled
![image](https://github.com/user-attachments/assets/b8ecc859-8eaf-4911-ac35-54f0b182d76a)

![image](https://github.com/user-attachments/assets/45f055ed-0295-49fc-b64c-a1a7e4bbcb65)

### What did you expect to see?

CPU utilization should come down to 2-3% when data is not being streamed

### What did you see instead?

CPU utilization is high even when data is not being streamed

### Anything else?

This is seen only when batching is enabled. When batching is disabled it works as expected.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the minimal reproduction described in the issue on Pulsar 2.10.6: stream data to 1,500 topics with batching enabled, stop the stream, and compare CPU usage with batching disabled. No source files or tests are named; done means identifying and fixing the batching-related post-stream CPU activity so utilization falls to the expected 2–3%.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.