apache / apache/bookkeeper

AutoRecovery caused DirectMemory OOM error.

Open
#3,408 10 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**Bookkeeper Version: 4.14.4**

Our prod environment went wrong last week, all bookies were killed because of direct memory OOM, this happened after one bookie's disk was broken, we tried to offline this bookie. After auditBookie triggered, all the bookies Direct Memory keep increase, it seem that, there is memory leak problem.

The ReplicateWorker log: x.x.x.x is the ip of lost bookie
```
022-07-14 21:50:41.721 [ReplicationWorker] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: null/x.x.x.x:3181, current state CONNECTING :
2022-07-14 21:50:41.723 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1502 from bookie: x.x.x.x:3181
2022-07-14 21:50:41.724 [ReplicationWorker] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:41.724 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1506 from bookie: x.x.x.x:3181
2022-07-14 21:50:41.724 [ReplicationWorker] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:41.724 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1510 from bookie: x.x.x.x:3181
2022-07-14 21:50:41.725 [ReplicationWorker] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:41.725 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1514 from bookie: x.x.x.x:3181
2022-07-14 21:50:41.725 [ReplicationWorker] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:41.725 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1518 from bookie: x.x.x.x:3181
2022-07-14 21:50:41.725 [ReplicationWorker] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running

2022-07-14 21:50:42.403 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1974 from bookie: x.x.x.x:3181
2022-07-14 21:50:42.440 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1978 from bookie: x.x.x.x:3181
2022-07-14 21:50:42.525 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1982 from bookie: x.x.x.x:3181
2022-07-14 21:50:42.593 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1986 from bookie: x.x.x.x:3181
2022-07-14 21:50:42.665 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1990 from bookie: x.x.x.x:3181
2022-07-14 21:50:42.706 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1994 from bookie: x.x.x.x:3181
2022-07-14 21:50:42.776 [BookKeeperClientWorker-OrderedExecutor-8-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie handle is not available while reading L61080496 E1998 from bookie: x.x.x.x:3181
2022-07-14 21:50:44.271 [BookKeeperClientWorker-OrderedExecutor-8-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:44.271 [BookKeeperClientWorker-OrderedExecutor-8-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:44.271 [BookKeeperClientWorker-OrderedExecutor-8-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:44.271 [BookKeeperClientWorker-OrderedExecutor-8-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:44.271 [BookKeeperClientWorker-OrderedExecutor-8-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running
2022-07-14 21:50:44.271 [BookKeeperClientWorker-OrderedExecutor-8-0] ERROR org.apache.bookkeeper.proto.PerChannelBookieClient - Cannot connect to x.x.x.x:3181 as endpoint resolution failed (probably bookie is down) err org.apache.bookkeeper.proto.BookieAddressResolver$BookieIdNotResolvedException: Cannot resolve bookieId x.x.x.x:3181, bookie does not exist or it is not running

2022-07-14 22:10:19.830 [BookKeeperClientWorker-OrderedExecutor-41-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L60558419 E359 from bookie: 10.71.168.13:3181
2022-07-14 22:10:19.830 [BookKeeperClientWorker-OrderedExecutor-41-0] ERROR org.apache.bookkeeper.client.LedgerFragmentReplicator - BK error reading ledger entry: 434
2022-07-14 22:10:19.831 [BookKeeperClientWorker-OrderedExecutor-41-0] ERROR org.apache.bookkeeper.proto.BookkeeperInternalCallbacks - Error in multi callback : -23

is (-1, rc = null)
2022-07-14 22:10:19.830 [BookKeeperClientWorker-OrderedExecutor-41-0] INFO org.apache.bookkeeper.client.PendingReadOp - Error: Bookie operation timeout while reading L60558419 E378 from bookie: 1.1.1.1:3181
2022-07-14 22:10:19.830 [BookKeeperClientWorker-OrderedExecutor-41-0] ERROR org.apache.bookkeeper.client.PendingReadOp - Read of ledger entry failed: L60558419 E378-E378, Sent to [x.x.x.x:3181, 1.1.1.1:3181], Heard from [] : bitset = {}, Error = 'Bookie operation timeout'. First unread entry is (-1, rc = null)

```

And there are bookies quarantined by brokers continuous, all bookies are crashed at last.

Note: autoRecoveryDaemonEnabled=true, we run autoRecovery service inside the bookie service.
After the bookie was lost because of disk failure, we also run a decommission command to offline this bookie. (I thinks it s a duplicate trigger of auditing bookie)

![image](https://user-images.githubusercontent.com/9278488/179349852-0128b1c1-b3df-46ce-a09f-ca348252093c.png)

NUM_ENTRIES_UNABLE_TO_READ_FOR_REPLICATION metrics
![image](https://user-images.githubusercontent.com/9278488/179452441-850ce615-e8d9-4fb1-a459-c97947ccb3c3.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.