apache / apache/pulsar

[Bug] With small entry sizes and high throughput, it's necessary to increase dispatcherMaxReadBatchSize and dispatcherMaxReadSizeBytes to utilize CPU on the broker

Open
#24,695 3 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

Experimental branch from current master

Test to reproduce is
https://github.com/lhotari/pulsar/blob/a85ac6fbfb3b763f7eaa4f44e232536748119ca4/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java#L124-L143

In the commit, the problem is fixed. Repo instructions will be later.

### Issue Description

The test creates small entries with only 128 bytes for each entry to stress entry creation.
Batching is turned off. Consuming is very slow with default settings and consuming cannot keep up with producing that is over 350000 msgs/s (on Mac M3 Max running Docker with OrbStack).
The consuming speed is only about 11000 msgs/s.

### Error messages

```text
No errors
```

### Reproducing the issue

The issue can be reproduced with https://github.com/lhotari/pulsar/blob/a85ac6fbfb3b763f7eaa4f44e232536748119ca4/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java and changes in the branch https://github.com/lhotari/pulsar/tree/lh-small-entry-size-performance-issue .

Please notice that the problem in the branch is fixed. The problem can be reproduced by reverting the last commit https://github.com/lhotari/pulsar/commit/a85ac6fbfb3b763f7eaa4f44e232536748119ca4

This "fixed" the performance issue:
```patch
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java
index a9f718103bf2f..a8d2ec3a5e32f 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java
@@ -219,7 +219,8 @@ protected PulsarClusterSpec.PulsarClusterSpecBuilder beforeSetupCluster(String c
brokerEnvs.put("dispatcherRetryBackoffMaxTimeInMs", "0");
//brokerEnvs.put("PULSAR_PREFIX_subscriptionKeySharedUseClassicPersistentImplementation", "true");
//brokerEnvs.put("PULSAR_PREFIX_subscriptionSharedUseClassicPersistentImplementation", "true");
- brokerEnvs.put("dispatcherMaxReadBatchSize", "1000");
+ brokerEnvs.put("dispatcherMaxReadBatchSize", "10000");
+ brokerEnvs.put("dispatcherMaxReadSizeBytes", "10000000");
brokerEnvs.put("dispatcherDispatchMessagesInSubscriptionThread", "false");
brokerEnvs.put("dispatcherMaxRoundRobinBatchSize", "1000");
specBuilder.brokerEnvs(brokerEnvs);
```

### Additional information

Although the changein the repro branch "fixed" the issue, it's not a suitable fix. The root cause of the problem should be addressed. It's possible that it's related to entry size estimation changes that have been made fairly recently.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java and reproduce the throughput gap after reverting commit a85ac6fbfb3b763f7eaa4f44e232536748119ca4. Investigate the dispatcherMaxReadBatchSize and dispatcherMaxReadSizeBytes behavior alongside the recently changed entry-size estimation. Done means the underlying cause is corrected rather than masked by larger dispatcher settings, with the profiling test demonstrating that small entries can be consumed at the expected throughput.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.