Rebalancing functions does nothing
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
Hitting the endpoint to rebalance functions does not appear to work consistently in Pulsar 2.7.2.
**To Reproduce**
Steps to reproduce the behavior:
First, we look at the function assignments:
```
$ curl fab08.example.domain.com:8080/admin/v2/worker/assignments -H "Authorization: Bearer eyJhb...mNog" | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1723 100 1723 0 0 210k 0 --:--:-- --:--:-- --:--:-- 210k
{
"c-pulsar-pcdc1-green-test-fw-fab09.example.domain.com-8080": [
"amplitude/processing/random-2:5",
"amplitude/processing/random-2:4",
"amplitude/processing/random-2:3",
"amplitude/processing/random-2:2",
"amplitude/processing/random-2:9",
"amplitude/processing/random-2:8",
"amplitude/processing/random-2:7",
"amplitude/processing/random-2:6",
"amplitude/processing/random-2:1",
"amplitude/processing/random-2:0",
"amplitude/processing/random-1:23",
"amplitude/processing/random-1:21",
"amplitude/processing/random-1:22",
"amplitude/processing/random-1:20",
"amplitude/processing/random-1:14",
"amplitude/processing/random-1:6",
"amplitude/processing/random-1:5",
"amplitude/processing/random-1:15",
"amplitude/processing/random-1:4",
"amplitude/processing/random-1:12",
"amplitude/processing/random-1:13",
"amplitude/processing/random-1:3",
"amplitude/processing/random-2:23",
"amplitude/processing/random-1:10",
"amplitude/processing/random-2:22",
"amplitude/processing/random-1:11",
"amplitude/processing/random-1:9",
"amplitude/processing/random-2:21",
"amplitude/processing/random-1:8",
"amplitude/processing/random-2:20",
"amplitude/processing/random-1:7",
"amplitude/processing/random-1:18",
"amplitude/processing/random-1:19",
"amplitude/processing/random-1:16",
"amplitude/processing/random-1:17",
"amplitude/processing/random-1:2",
"amplitude/processing/random-1:1",
"amplitude/processing/random-1:0",
"amplitude/processing/random-2:16",
"amplitude/processing/random-2:15",
"amplitude/processing/random-2:14",
"amplitude/processing/random-2:13",
"amplitude/processing/random-2:12",
"amplitude/processing/random-2:11",
"amplitude/processing/random-2:10",
"amplitude/processing/random-2:19",
"amplitude/processing/random-2:18",
"amplitude/processing/random-2:17"
]
}
```
Next, we trigger functions to rebalance:
`$ curl fab08.example.domain.com:8080/admin/v2/worker/rebalance -X PUT -H "Authorization: Bearer eyJ...mNog"`
Checking function assignments again after a few minutes shows no changes, as demonstrated below:
```
$ curl fab08.example.domain.com:8080/admin/v2/worker/assignments -H "Authorization: Bearer eyJ...mNog" | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1723 100 1723 0 0 560k 0 --:--:-- --:--:-- --:--:-- 560k
{
"c-pulsar-pcdc1-green-test-fw-fab09.example.domain.com-8080": [
"amplitude/processing/random-2:5",
"amplitude/processing/random-2:4",
"amplitude/processing/random-2:3",
"amplitude/processing/random-2:2",
"amplitude/processing/random-2:9",
"amplitude/processing/random-2:8",
"amplitude/processing/random-2:7",
"amplitude/processing/random-2:6",
"amplitude/processing/random-2:1",
"amplitude/processing/random-2:0",
"amplitude/processing/random-1:23",
"amplitude/processing/random-1:21",
"amplitude/processing/random-1:22",
"amplitude/processing/random-1:20",
"amplitude/processing/random-1:14",
"amplitude/processing/random-1:6",
"amplitude/processing/random-1:5",
"amplitude/processing/random-1:15",
"amplitude/processing/random-1:4",
"amplitude/processing/random-1:12",
"amplitude/processing/random-1:13",
"amplitude/processing/random-1:3",
"amplitude/processing/random-2:23",
"amplitude/processing/random-1:10",
"amplitude/processing/random-2:22",
"amplitude/processing/random-1:11",
"amplitude/processing/random-1:9",
"amplitude/processing/random-2:21",
"amplitude/processing/random-1:8",
"amplitude/processing/random-2:20",
"amplitude/processing/random-1:7",
"amplitude/processing/random-1:18",
"amplitude/processing/random-1:19",
"amplitude/processing/random-1:16",
"amplitude/processing/random-1:17",
"amplitude/processing/random-1:2",
"amplitude/processing/random-1:1",
"amplitude/processing/random-1:0",
"amplitude/processing/random-2:16",
"amplitude/processing/random-2:15",
"amplitude/processing/random-2:14",
"amplitude/processing/random-2:13",
"amplitude/processing/random-2:12",
"amplitude/processing/random-2:11",
"amplitude/processing/random-2:10",
"amplitude/processing/random-2:19",
"amplitude/processing/random-2:18",
"amplitude/processing/random-2:17"
]
}
```
After some experimentation, I discovered that I was able to trigger rebalancing to occur if I targeted the function worker leader, but it's not clear if this happens consistently or not.
**Expected behavior**
Triggering function rebalancing should work consistently when triggered on any broker.
Also, if there is a failure, it should be reported in the logs. In the current implementation, when sending the rebalance request, no logs appeared in the targeted broker except when it succeeded.
More logging should indicate if there's a problem on the broker that receives the signal to rebalance functions.
Contributor guide
Research direction
Start with the PUT /admin/v2/worker/rebalance endpoint and the worker assignment endpoint described in the report. Reproduce the request against different brokers, including the function worker leader, while reviewing broker logs for failures. Done means rebalancing is triggered consistently from any broker and failures are logged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100