[Bug] AutoClusterFailover method probeAvailable() always return true if the brokers are behind loadbalancer
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Version
OS: Linux
Pulsar client version: 2.11.1
Pulsar broker version: 2.9.3
### Minimal reproduce step
1/ Deploy two pulsar broker clusters with geo-replication enabled on Kubernetes (AKS) with Istio
2/ Configure Istio virtual service to expose pulsar-proxy service (load balancer)
3/ A consumer utilizes auto-failover pulsar client to consume messages from pulsar brokers
4/ While the consumer is consuming from the primary broker, scale down pulsar-proxy service on the primary broker
### What did you expect to see?
The consumer should consume from secondary broker
### What did you see instead?
The consumer did not switch to the secondary broker as the probeAvailable() method returns true because the load balancer is still up
### Anything else?
I think it would be better if we change to call pulsar admin API for probeAvailable() method instead of socket.connect()
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start by locating the AutoClusterFailover client code and its probeAvailable() method, then trace how socket.connect() determines broker availability behind the load balancer. Reproduce the primary proxy scale-down scenario and verify that the consumer switches to the secondary broker without treating the still-up load balancer as broker availability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100