apache / apache/pulsar

[Bug] Bookkeeper rackaware placement policy not applying on bookies

Open
#20,918 19 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [ ] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
A bit similar: https://github.com/apache/pulsar/issues/3716

### Version

Pulsar version: 3.0.0

### Minimal reproduce step

We are trying to mitigate the following warning message from bookkeeper and attempted to set the RackAware policy on the bookies, but we couldn't make it to work.
`New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy. quarantinedBookies: []`

Setup running in AWS:
3 Bookkeeper instance on each availability zone (us-east-1a, us-east-1b, us-east-1c)
3 Zookeeper (us-east-1a, us-east-1b, us-east-1c)
2 Broker (us-east-1a, us-east-1b)
2 Proxy (us-east-1b, us-east-1c)

Bookies:
```
[root@ip-172-16-1-57 pulsar]# bin/pulsar-admin bookies list-bookies
{
"bookies" : [ {
"bookieId" : "172.16.1.112:3181"
}, {
"bookieId" : "172.16.2.190:3181"
}, {
"bookieId" : "172.16.3.6:3181"
} ]
}
```

Set up racks placement with the following command:
```
$PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.2.190:3181 --group test --hostname 172.16.2.190:3181 --rack /us-east-1a
$PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.1.112:3181 --group test --hostname 172.16.1.112:3181 --rack /us-east-1b
$PULSAR_HOME/bin/pulsar-admin bookies set-bookie-rack --bookie 172.16.3.6:3181 --group test --hostname 172.16.3.6:3181 --rack /us-east-1c
```

Get the rack placement for the bookies:
```
[root@ip-172-16-1-57 pulsar]# bin/pulsar-admin bookies racks-placement
test
{
172.16.2.190:3181=BookieInfoImpl(rack=/us-east-1a, hostname=172.16.2.190:3181),
172.16.1.112:3181=BookieInfoImpl(rack=/us-east-1b, hostname=172.16.1.112:3181),
172.16.3.6:3181=BookieInfoImpl(rack=/us-east-1c, hostname=172.16.3.6:3181)
}
```

### What did you expect to see?

Something in the logs that tells the policy applied on the bookies.

### What did you see instead?

Nothing new, only the this warning again, like 50 times:
`New ensemble: [172.16.1.112:3181, 172.16.3.6:3181, 172.16.2.190:3181] is not adhering to Placement Policy. quarantinedBookies: []`

### Anything else?

Configs:
[broker_conf.txt](https://github.com/apache/pulsar/files/12238023/broker_conf.txt)
[bookkeeper_conf.txt](https://github.com/apache/pulsar/files/12238025/bookkeeper_conf.txt)

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reviewing broker_conf.txt and bookkeeper_conf.txt, then reproduce the setup with the shown set-bookie-rack and racks-placement commands. Trace how the configured racks relate to the repeated “not adhering to Placement Policy” warning, and confirm the issue is resolved when the rack-aware policy is actually applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.