[Bug] With small entry sizes and high throughput, it's necessary to increase dispatcherMaxReadBatchSize and dispatcherMaxReadSizeBytes to utilize CPU on the broker
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### User environment
Experimental branch from current master
Test to reproduce is
https://github.com/lhotari/pulsar/blob/a85ac6fbfb3b763f7eaa4f44e232536748119ca4/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java#L124-L143
In the commit, the problem is fixed. Repo instructions will be later.
### Issue Description
The test creates small entries with only 128 bytes for each entry to stress entry creation.
Batching is turned off. Consuming is very slow with default settings and consuming cannot keep up with producing that is over 350000 msgs/s (on Mac M3 Max running Docker with OrbStack).
The consuming speed is only about 11000 msgs/s.
### Error messages
```text
No errors
```
### Reproducing the issue
The issue can be reproduced with https://github.com/lhotari/pulsar/blob/a85ac6fbfb3b763f7eaa4f44e232536748119ca4/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java and changes in the branch https://github.com/lhotari/pulsar/tree/lh-small-entry-size-performance-issue .
Please notice that the problem in the branch is fixed. The problem can be reproduced by reverting the last commit https://github.com/lhotari/pulsar/commit/a85ac6fbfb3b763f7eaa4f44e232536748119ca4
This "fixed" the performance issue:
```patch
diff --git a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java
index a9f718103bf2f..a8d2ec3a5e32f 100644
--- a/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java
+++ b/tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java
@@ -219,7 +219,8 @@ protected PulsarClusterSpec.PulsarClusterSpecBuilder beforeSetupCluster(String c
brokerEnvs.put("dispatcherRetryBackoffMaxTimeInMs", "0");
//brokerEnvs.put("PULSAR_PREFIX_subscriptionKeySharedUseClassicPersistentImplementation", "true");
//brokerEnvs.put("PULSAR_PREFIX_subscriptionSharedUseClassicPersistentImplementation", "true");
- brokerEnvs.put("dispatcherMaxReadBatchSize", "1000");
+ brokerEnvs.put("dispatcherMaxReadBatchSize", "10000");
+ brokerEnvs.put("dispatcherMaxReadSizeBytes", "10000000");
brokerEnvs.put("dispatcherDispatchMessagesInSubscriptionThread", "false");
brokerEnvs.put("dispatcherMaxRoundRobinBatchSize", "1000");
specBuilder.brokerEnvs(brokerEnvs);
```
### Additional information
Although the changein the repro branch "fixed" the issue, it's not a suitable fix. The root cause of the problem should be addressed. It's possible that it's related to entry size estimation changes that have been made fairly recently.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with tests/integration/src/test/java/org/apache/pulsar/tests/integration/profiling/PulsarProfilingTest.java and reproduce the throughput gap after reverting commit a85ac6fbfb3b763f7eaa4f44e232536748119ca4. Investigate the dispatcherMaxReadBatchSize and dispatcherMaxReadSizeBytes behavior alongside the recently changed entry-size estimation. Done means the underlying cause is corrected rather than masked by larger dispatcher settings, with the profiling test demonstrating that small entries can be consumed at the expected throughput.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100