apache / apache/pulsar

Stop one broker may cause another broker shutdown when enable function woker in broker

Open
#11,695 1 comment 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**To Reproduce**
1. Start pulsar cluster with 3 brokers and `functionsWorkerEnabled=true`
2. Stop the broker which is the leader of function worker and the owner of function worker coordinate topic.
3. Another broker will become leader and then shutdown due to fail to get coordinate topic.

**Expected behavior**
Should not trigger shutdown when fail to get coordinate topic.

**Additional context**
When stop the broker which is the leader of function woker and the owner of coordinate topic. Another broker will become leader but coordinate topic did not shift to another broker in time.
There is a schedule task runing only on leader and will get follow error:

> ERROR org.apache.pulsar.functions.worker.ErrorNotifierShutdownServiceImpl - Encountered fatal error. Shutting down.
> java.lang.RuntimeException: org.apache.pulsar.client.admin.PulsarAdminException: java.util.concurrent.CompletionException: org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException: Could not complete the operation. Number of retries has been exhausted. Failed reason: Connection refused: /xxx.xxx.xxx.xxx:8080
> at org.apache.pulsar.functions.worker.MembershipManager.getCurrentMembership(MembershipManager.java:78)

`getCurrentMembership` will throw a `RuntimeException` when fail to get coordinate topic. And because we are using `ErrorNotifierShutdownServiceImpl` the broker will shutdown when get exception.

There may be two parts need to be improved:
1. Add error handle when get coordinate topic.
2. Do not use `ErrorNotifierShutdownServiceImpl`, it is not suitable to shutdown broker when function worker get any exception.

Contributor guide

Open the contributing guide

Research direction

Start at MembershipManager.getCurrentMembership in MembershipManager.java:78 and trace the leader-only schedule task and ErrorNotifierShutdownServiceImpl. Reproduce the three-broker failover with functionsWorkerEnabled=true, then verify that a temporary coordinate-topic lookup failure does not shut down the newly elected broker.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.