apache / apache/pulsar

[Bug] AutoClusterFailover method probeAvailable() always return true if the brokers are behind loadbalancer

Open
#20,399 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

OS: Linux
Pulsar client version: 2.11.1
Pulsar broker version: 2.9.3

### Minimal reproduce step

1/ Deploy two pulsar broker clusters with geo-replication enabled on Kubernetes (AKS) with Istio
2/ Configure Istio virtual service to expose pulsar-proxy service (load balancer)
3/ A consumer utilizes auto-failover pulsar client to consume messages from pulsar brokers
4/ While the consumer is consuming from the primary broker, scale down pulsar-proxy service on the primary broker

### What did you expect to see?

The consumer should consume from secondary broker

### What did you see instead?

The consumer did not switch to the secondary broker as the probeAvailable() method returns true because the load balancer is still up

### Anything else?

I think it would be better if we change to call pulsar admin API for probeAvailable() method instead of socket.connect()

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by locating the AutoClusterFailover client code and its probeAvailable() method, then trace how socket.connect() determines broker availability behind the load balancer. Reproduce the primary proxy scale-down scenario and verify that the consumer switches to the secondary broker without treating the still-up load balancer as broker availability.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.