apache / apache/pulsar-client-go

receiverQueueSize in case of partitioned topics

Open
#466 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
745
Forks
389
Avg merge
3d 20h
Merged PRs (30d)
3

Description

**Is your feature request related to a problem? Please describe.**
Currently consumer option receiverQueueSize defines two things:
1. the size of the permits we send to broker
2. the size of the channel that receives messages inside consumer

When we are using partitioned topics, we create separate instance of partitionConsumer for every partition.
That leads to the situation when each partitioned consumer sends it's own flow command to the broker with the `receiverQueueSize` permits.

As a result total number of permits for consumer is equal to `receiverQueueSize * numberOfTopicPartitions`.

For the other side size of messages channel inside consumer would be just of `receiverQueueSize`.

So broker would expect to send much more messages to the consumers than it could process.

**Describe the solution you'd like**
Probably we must set `partitionConsumerOpts.receiverQueueSize` to be equal to `receiverQueueSize / numberOfTopicPartitions` or make it able to configure it independently from receiverQueueSize.

Special case when we have receiverQueueSize equal to 1. How could we handle this situation?

Contributor guide

Open the contributing guide

Research direction

Start by locating the receiverQueueSize and partitionConsumerOpts handling in the Go client, then trace how partition consumers send permits and how the shared message channel is sized. The design should account for partitioned topics and receiverQueueSize equal to 1, with broker permits consistent with the consumer's processing capacity.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.