Feature Request: Add option for isolated connections for Pulsar consumers and producers
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Motivation
Currently, in Pulsar, TCP connections are shared across all consumers and producers for a client. This can lead to issues with backpressure and flow control, particularly for producers. The shared connection usage causes [head-of-line blocking](https://en.wikipedia.org/wiki/Head-of-line_blocking) problems. To address this, it is proposed to add configuration options to allow consumers and producers to use isolated TCP/IP connections or connection groups.
- Allow configuring individual consumers or producers to use a dedicated, non-shared TCP connection (or connection group) to the broker.
- Provide an option for multi-partition producers to allow sharing of the dedicated connections for all partitions of that producer instance.
- Implement similar isolation options for consumers.
- This feature would complement the proposed throttling-based flow control solution in "PIP-385: Add rate limit semantics to Pulsar protocol and Java Client" #23398, providing an additional mechanism to improve connection management and reduce interference between different producers/consumers.
- The isolation options would be configurable through the client API, allowing developers to fine-tune connection usage based on their specific requirements.
### Benefits
- Reduced interference between producers/consumers sharing connections
- More predictable performance for high-priority or high-volume producers/consumers
- Easier troubleshooting of connection-related issues
This feature provides a simple improvement in connection management and performance isolation, which can be implemented as an interim solution or alongside more comprehensive flow control mechanisms such as PIP-385.
### Additional Context
- "PIP-385: Add rate limit semantics to Pulsar protocol and Java Client" #23398
Contributor guide
Assessment
This issue has not been assessed yet.