apache / apache/pulsar

Rebalancing functions does nothing

Open
#12,928 3 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
Hitting the endpoint to rebalance functions does not appear to work consistently in Pulsar 2.7.2.

**To Reproduce**
Steps to reproduce the behavior:

First, we look at the function assignments:

```
$ curl fab08.example.domain.com:8080/admin/v2/worker/assignments -H "Authorization: Bearer eyJhb...mNog" | python -m json.tool

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1723 100 1723 0 0 210k 0 --:--:-- --:--:-- --:--:-- 210k

{

"c-pulsar-pcdc1-green-test-fw-fab09.example.domain.com-8080": [
"amplitude/processing/random-2:5",
"amplitude/processing/random-2:4",
"amplitude/processing/random-2:3",
"amplitude/processing/random-2:2",
"amplitude/processing/random-2:9",
"amplitude/processing/random-2:8",
"amplitude/processing/random-2:7",
"amplitude/processing/random-2:6",
"amplitude/processing/random-2:1",
"amplitude/processing/random-2:0",
"amplitude/processing/random-1:23",
"amplitude/processing/random-1:21",
"amplitude/processing/random-1:22",
"amplitude/processing/random-1:20",
"amplitude/processing/random-1:14",
"amplitude/processing/random-1:6",
"amplitude/processing/random-1:5",
"amplitude/processing/random-1:15",
"amplitude/processing/random-1:4",
"amplitude/processing/random-1:12",
"amplitude/processing/random-1:13",
"amplitude/processing/random-1:3",
"amplitude/processing/random-2:23",
"amplitude/processing/random-1:10",
"amplitude/processing/random-2:22",
"amplitude/processing/random-1:11",
"amplitude/processing/random-1:9",
"amplitude/processing/random-2:21",
"amplitude/processing/random-1:8",
"amplitude/processing/random-2:20",
"amplitude/processing/random-1:7",
"amplitude/processing/random-1:18",
"amplitude/processing/random-1:19",
"amplitude/processing/random-1:16",
"amplitude/processing/random-1:17",
"amplitude/processing/random-1:2",
"amplitude/processing/random-1:1",
"amplitude/processing/random-1:0",
"amplitude/processing/random-2:16",
"amplitude/processing/random-2:15",
"amplitude/processing/random-2:14",
"amplitude/processing/random-2:13",
"amplitude/processing/random-2:12",
"amplitude/processing/random-2:11",
"amplitude/processing/random-2:10",
"amplitude/processing/random-2:19",
"amplitude/processing/random-2:18",
"amplitude/processing/random-2:17"
]

}
```

Next, we trigger functions to rebalance:

`$ curl fab08.example.domain.com:8080/admin/v2/worker/rebalance -X PUT -H "Authorization: Bearer eyJ...mNog"`

Checking function assignments again after a few minutes shows no changes, as demonstrated below:

```
$ curl fab08.example.domain.com:8080/admin/v2/worker/assignments -H "Authorization: Bearer eyJ...mNog" | python -m json.tool

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1723 100 1723 0 0 560k 0 --:--:-- --:--:-- --:--:-- 560k

{

"c-pulsar-pcdc1-green-test-fw-fab09.example.domain.com-8080": [
"amplitude/processing/random-2:5",
"amplitude/processing/random-2:4",
"amplitude/processing/random-2:3",
"amplitude/processing/random-2:2",
"amplitude/processing/random-2:9",
"amplitude/processing/random-2:8",
"amplitude/processing/random-2:7",
"amplitude/processing/random-2:6",
"amplitude/processing/random-2:1",
"amplitude/processing/random-2:0",
"amplitude/processing/random-1:23",
"amplitude/processing/random-1:21",
"amplitude/processing/random-1:22",
"amplitude/processing/random-1:20",
"amplitude/processing/random-1:14",
"amplitude/processing/random-1:6",
"amplitude/processing/random-1:5",
"amplitude/processing/random-1:15",
"amplitude/processing/random-1:4",
"amplitude/processing/random-1:12",
"amplitude/processing/random-1:13",
"amplitude/processing/random-1:3",
"amplitude/processing/random-2:23",
"amplitude/processing/random-1:10",
"amplitude/processing/random-2:22",
"amplitude/processing/random-1:11",
"amplitude/processing/random-1:9",
"amplitude/processing/random-2:21",
"amplitude/processing/random-1:8",
"amplitude/processing/random-2:20",
"amplitude/processing/random-1:7",
"amplitude/processing/random-1:18",
"amplitude/processing/random-1:19",
"amplitude/processing/random-1:16",
"amplitude/processing/random-1:17",
"amplitude/processing/random-1:2",
"amplitude/processing/random-1:1",
"amplitude/processing/random-1:0",
"amplitude/processing/random-2:16",
"amplitude/processing/random-2:15",
"amplitude/processing/random-2:14",
"amplitude/processing/random-2:13",
"amplitude/processing/random-2:12",
"amplitude/processing/random-2:11",
"amplitude/processing/random-2:10",
"amplitude/processing/random-2:19",
"amplitude/processing/random-2:18",
"amplitude/processing/random-2:17"
]

}
```

After some experimentation, I discovered that I was able to trigger rebalancing to occur if I targeted the function worker leader, but it's not clear if this happens consistently or not.

**Expected behavior**
Triggering function rebalancing should work consistently when triggered on any broker.
Also, if there is a failure, it should be reported in the logs. In the current implementation, when sending the rebalance request, no logs appeared in the targeted broker except when it succeeded.
More logging should indicate if there's a problem on the broker that receives the signal to rebalance functions.

Contributor guide

Open the contributing guide

Research direction

Start with the PUT /admin/v2/worker/rebalance endpoint and the worker assignment endpoint described in the report. Reproduce the request against different brokers, including the function worker leader, while reviewing broker logs for failures. Done means rebalancing is triggered consistently from any broker and failures are logged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.