apache / apache/pulsar

[Question][scalable-topics] How should controller subscribe be routed through a proxy?

Open
#26,275 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

## Context

Without a proxy, `ScalableConsumerClient` connects to the `controller_broker_url` returned by the temporary DAG lookup. If that URL is absent, it falls back to the service URL.

Behind a proxy, the Java client calls `getAnyBrokerProxyConnection()`. The empty `proxy_to_broker_url` sentinel makes the proxy select an arbitrary broker and then transparently bridge bytes to it.

On the broker side, scalable subscribe appears to call `getOrCreateController()` and then a leader-only `registerConsumer()` guarded by `checkLeader()`. I could not find forwarding from a non-leader broker to the elected controller. This suggests that proxy-any-broker succeeds only when the arbitrary broker is or becomes controller leader.

The existing proxy scalable-topic test uses `QueueConsumer`, which never sends controller subscribe, so it does not exercise this path. The multi-broker StreamConsumer test waits for controller state to converge and exercises the explicit controller URL path.

## Questions

1. Is `CommandScalableTopicSubscribe` intended to be accepted or forwarded by any broker, or only by the elected controller?
2. If only the controller accepts it, should the proxy pair directly to the advertised controller instead of selecting any broker?
3. What should clients do while `controller_broker_url` is absent?
4. Should DAG watches emit an update when controller leadership changes without a layout change?
5. Which URL/listener should TLS and advertised-listener clients use for controller registration?
6. Is a real proxy StreamConsumer test with assignment pushes and reconnect planned?

References: PIP-468, PR #25879, `ScalableConsumerClient`, `ProxyConnection`, `ScalableTopicService`, `ScalableTopicController`, `ProxyScalableTopicsTest`, and `V5MultiBrokerScalableTopicTest`.

Contributor guide

Open the contributing guide

Research direction

Start with ScalableConsumerClient, ProxyConnection, ScalableTopicService, and ScalableTopicController, then compare ProxyScalableTopicsTest with V5MultiBrokerScalableTopicTest. Trace controller subscription through proxy and leadership changes, using PIP-468 and PR #25879 as context. Done requires an agreed routing and client-behavior contract, including TLS and reconnect cases, plus representative proxy StreamConsumer coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.