apache / apache/pulsar

[Bug] Permit calculations are mixing individual message and batch message counts, similar issue in documentation

Open
#23,263 2 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [X] I understand that unsupported versions don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### Version

master

### Minimal reproduce step

Fixing some issues as part of #23231 such as
https://github.com/apache/pulsar/blob/14395295b4996dcfb7eac288d92baf1104c9c576/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentDispatcherMultipleConsumers.java#L790-L791

The problem is that permits use a batch message count and unack messages uses individual message counts. These shouldn't be mixed. It's possible to convert the batch message count to an estimated number of messages with the average number of messages in batch value that is kept in the consumer.

There are several locations where this is mixed up.

This also applies to documentation. The javadoc of receiverQueueSize is very confusing and doesn't tell that it's counted in batch messages: https://github.com/apache/pulsar/blob/10f4e0248f0f985b1dc7ad38970c906b7fe629be/pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java#L356-L387

### What did you expect to see?

The permit calculations in dispatchers shouldn't mix individual message counts and batch message counts.
The documentation needs updates too so that it's clearly defined whether a "message" refers to an individual message or a batch message (a batch of individual messages).

### What did you see instead?

- calculations mixed in several locations
- documentation doesn't clearly define whether a message is an individual message or a batch of messages. example is the receiverQueueSize javadoc.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with PersistentDispatcherMultipleConsumers.java at the linked permit calculation and inspect the other locations where individual and batch message counts are mixed. Then read the receiverQueueSize Javadoc in pulsar-client-api/src/main/java/org/apache/pulsar/client/api/ConsumerBuilder.java; done means calculations use consistent count semantics and the documentation clearly distinguishes individual messages from batches.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.