apache / apache/bookkeeper

Deadlock occurred when entryLogPerLedgerEnabled is true

Open
#3,780 4 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**BUG REPORT**

Recently I'm doing benchmark on bookie with entryLogPerLedgerEnabled = true, I found the process happened deadlock and hence cann't service normally, here is the threaddump information while error occured.

![image](https://user-images.githubusercontent.com/32755366/218699202-40e71c5f-6881-464b-b4a6-8fade94ca871.png)
![image](https://user-images.githubusercontent.com/32755366/218699248-bd1402cc-4ebc-4d3f-b1a4-e0b632688c4f.png)

Bookkeer Versoin: 4.15.0

Here is our customize configuration
`ledgerStorageClass=org.apache.bookkeeper.bookie.SortedLedgerStorage`
`entryLogPerLedgerEnabled=true`
`maximumNumberOfActiveEntryLogs=10000`

It seems caused by acquire lock while EntryMemTableParallelFlusher is running , and the deadlock may happen like this

1. Memtable reach size limit
2. ledger A and ledger B start flush asynchronous, and they both acquire lock successfully
3. when thread enters getCurrentLogWithDirInfoForLedger method, it will calls ledgerIdEntryLogMap#get(ledgerId), if ledgerIdEntryLogMap is out of capacity and trigger cleanup ledger A、B, thus onCacheEntryRemoval is invoked, this method attempt to acquire lock , Note if thread A(thread which run ledger A) exactly execute cleanup ledger B ,and thread B execute cleanup ledger A, deadlock is happened.

image

image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.