apache / apache/pulsar-client-go
receiverQueueSize in case of partitioned topics
- Dominant language
- Go
- Stars
- 745
- Forks
- 389
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 3
Description
**Is your feature request related to a problem? Please describe.**
Currently consumer option receiverQueueSize defines two things:
1. the size of the permits we send to broker
2. the size of the channel that receives messages inside consumer
When we are using partitioned topics, we create separate instance of partitionConsumer for every partition.
That leads to the situation when each partitioned consumer sends it's own flow command to the broker with the `receiverQueueSize` permits.
As a result total number of permits for consumer is equal to `receiverQueueSize * numberOfTopicPartitions`.
For the other side size of messages channel inside consumer would be just of `receiverQueueSize`.
So broker would expect to send much more messages to the consumers than it could process.
**Describe the solution you'd like**
Probably we must set `partitionConsumerOpts.receiverQueueSize` to be equal to `receiverQueueSize / numberOfTopicPartitions` or make it able to configure it independently from receiverQueueSize.
Special case when we have receiverQueueSize equal to 1. How could we handle this situation?
Contributor guide
Research direction
Start by locating the receiverQueueSize and partitionConsumerOpts handling in the Go client, then trace how partition consumers send permits and how the shared message channel is sized. The design should account for partitioned topics and receiverQueueSize equal to 1, with broker permits consistent with the consumer's processing capacity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100