Broker direct memory leak due to BKNotEnoughBookiesException
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
The benchmark test is for the max throughput scenario with 1GB/s message writing and 1GB/s message reading.
```
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=223240,loopId=5cca03e4) Exception updating
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:223240, change-id:0000000001)][attempt:1] Exception changing ensemble
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 223240 due to NotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=233784,loopId=0147ac3a) Exception updating
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:233784, change-id:0000000001)][attempt:1] Exception changing ensemble
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 233784 due to NotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=238581,loopId=3062cb0d) Exception updating
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:238581, change-id:0000000001)][attempt:1] Exception changing ensemble
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 238581 due to NotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,777+0000 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=259791,loopId=15db4235) Exception updating
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,777+0000 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:259791, change-id:0000000001)][attempt:1] Exception changing ensemble
```

The second broker does not have the above error logs.
**Additional context**
Build from branch-2.11 with bookkeeper 4.15.1
Contributor guide
Assessment
This issue has not been assessed yet.