apache / apache/pulsar

Broker direct memory leak due to BKNotEnoughBookiesException

Open
#16,998 5 comments 0 reactions 2 assignees Claimed by @Technoboy- View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**

The benchmark test is for the max throughput scenario with 1GB/s message writing and 1GB/s message reading.

```
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=223240,loopId=5cca03e4) Exception updating
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:223240, change-id:0000000001)][attempt:1] Exception changing ensemble
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 223240 due to NotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=233784,loopId=0147ac3a) Exception updating
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:233784, change-id:0000000001)][attempt:1] Exception changing ensemble
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 233784 due to NotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=238581,loopId=3062cb0d) Exception updating
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:238581, change-id:0000000001)][attempt:1] Exception changing ensemble
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 238581 due to NotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,777+0000 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=259791,loopId=15db4235) Exception updating
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available
Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 2022-08-08T17:44:23,777+0000 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:259791, change-id:0000000001)][attempt:1] Exception changing ensemble
```

image

The second broker does not have the above error logs.

**Additional context**
Build from branch-2.11 with bookkeeper 4.15.1

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.