[Question][scalable-topics] How should controller subscribe be routed through a proxy?
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
## Context
Without a proxy, `ScalableConsumerClient` connects to the `controller_broker_url` returned by the temporary DAG lookup. If that URL is absent, it falls back to the service URL.
Behind a proxy, the Java client calls `getAnyBrokerProxyConnection()`. The empty `proxy_to_broker_url` sentinel makes the proxy select an arbitrary broker and then transparently bridge bytes to it.
On the broker side, scalable subscribe appears to call `getOrCreateController()` and then a leader-only `registerConsumer()` guarded by `checkLeader()`. I could not find forwarding from a non-leader broker to the elected controller. This suggests that proxy-any-broker succeeds only when the arbitrary broker is or becomes controller leader.
The existing proxy scalable-topic test uses `QueueConsumer`, which never sends controller subscribe, so it does not exercise this path. The multi-broker StreamConsumer test waits for controller state to converge and exercises the explicit controller URL path.
## Questions
1. Is `CommandScalableTopicSubscribe` intended to be accepted or forwarded by any broker, or only by the elected controller?
2. If only the controller accepts it, should the proxy pair directly to the advertised controller instead of selecting any broker?
3. What should clients do while `controller_broker_url` is absent?
4. Should DAG watches emit an update when controller leadership changes without a layout change?
5. Which URL/listener should TLS and advertised-listener clients use for controller registration?
6. Is a real proxy StreamConsumer test with assignment pushes and reconnect planned?
References: PIP-468, PR #25879, `ScalableConsumerClient`, `ProxyConnection`, `ScalableTopicService`, `ScalableTopicController`, `ProxyScalableTopicsTest`, and `V5MultiBrokerScalableTopicTest`.
Contributor guide
Research direction
Start with ScalableConsumerClient, ProxyConnection, ScalableTopicService, and ScalableTopicController, then compare ProxyScalableTopicsTest with V5MultiBrokerScalableTopicTest. Trace controller subscription through proxy and leadership changes, using PIP-468 and PR #25879 as context. Done requires an agreed routing and client-behavior contract, including TLS and reconnect cases, plus representative proxy StreamConsumer coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100