Stop one broker may cause another broker shutdown when enable function woker in broker
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**To Reproduce**
1. Start pulsar cluster with 3 brokers and `functionsWorkerEnabled=true`
2. Stop the broker which is the leader of function worker and the owner of function worker coordinate topic.
3. Another broker will become leader and then shutdown due to fail to get coordinate topic.
**Expected behavior**
Should not trigger shutdown when fail to get coordinate topic.
**Additional context**
When stop the broker which is the leader of function woker and the owner of coordinate topic. Another broker will become leader but coordinate topic did not shift to another broker in time.
There is a schedule task runing only on leader and will get follow error:
> ERROR org.apache.pulsar.functions.worker.ErrorNotifierShutdownServiceImpl - Encountered fatal error. Shutting down.
> java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Connection refused: /xxx.xxx.xxx.xxx:8080
> at org.apache.pulsar.functions.worker.MembershipManager.getCurrentMembership(MembershipManager.java:78)
`getCurrentMembership` will throw a `RuntimeException` when fail to get coordinate topic. And because we are using `ErrorNotifierShutdownServiceImpl` the broker will shutdown when get exception.
There may be two parts need to be improved:
1. Add error handle when get coordinate topic.
2. Do not use `ErrorNotifierShutdownServiceImpl`, it is not suitable to shutdown broker when function worker get any exception.
Contributor guide
Research direction
Start at MembershipManager.getCurrentMembership in MembershipManager.java:78 and trace the leader-only schedule task and ErrorNotifierShutdownServiceImpl. Reproduce the three-broker failover with functionsWorkerEnabled=true, then verify that a temporary coordinate-topic lookup failure does not shut down the newly elected broker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100