[Bug] Bookkeeper rackaware placement policy not applying on bookies
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [ ] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
A bit similar: https://github.com/apache/pulsar/issues/3716
### Version
Pulsar version: 3.0.0
### Minimal reproduce step
We are trying to mitigate the following warning message from bookkeeper and attempted to set the RackAware policy on the bookies, but we couldn't make it to work.
`New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy. quarantinedBookies: []`
Setup running in AWS:
3 Bookkeeper instance on each availability zone (us-east-1a, us-east-1b, us-east-1c)
3 Zookeeper (us-east-1a, us-east-1b, us-east-1c)
2 Broker (us-east-1a, us-east-1b)
2 Proxy (us-east-1b, us-east-1c)
Bookies:
```
[root@ip-172-16-1-57 pulsar]# bin/pulsar-admin bookies list-bookies
{
"bookies" : [ {
"bookieId" : "172.16.1.112:3181"
}, {
"bookieId" : "172.16.2.190:3181"
}, {
"bookieId" : "172.16.3.6:3181"
} ]
}
```
Set up racks placement with the following command:
```
$PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.2.190:3181 --group test --hostname 172.16.2.190:3181 --rack /us-east-1a
$PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.1.112:3181 --group test --hostname 172.16.1.112:3181 --rack /us-east-1b
$PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.3.6:3181 --group test --hostname 172.16.3.6:3181 --rack /us-east-1c
```
Get the rack placement for the bookies:
```
[root@ip-172-16-1-57 pulsar]# bin/pulsar-admin bookies racks-placement
test
{
172.16.2.190:3181=BookieInfoImpl(rack=/us-east-1a, hostname=172.16.2.190:3181),
172.16.1.112:3181=BookieInfoImpl(rack=/us-east-1b, hostname=172.16.1.112:3181),
172.16.3.6:3181=BookieInfoImpl(rack=/us-east-1c, hostname=172.16.3.6:3181)
}
```
### What did you expect to see?
Something in the logs that tells the policy applied on the bookies.
### What did you see instead?
Nothing new, only the this warning again, like 50 times:
`New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy. quarantinedBookies: []`
### Anything else?
Configs:
[broker_conf.txt](https://github.com/apache/pulsar/files/12238023/broker_conf.txt)
[bookkeeper_conf.txt](https://github.com/apache/pulsar/files/12238025/bookkeeper_conf.txt)
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start by reviewing broker_conf.txt and bookkeeper_conf.txt, then reproduce the setup with the shown set-bookie-rack and racks-placement commands. Trace how the configured racks relate to the repeated “not adhering to Placement Policy” warning, and confirm the issue is resolved when the rack-aware policy is actually applied.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100