apache / apache/bookkeeper

the concurrency problem of the compaction process and the regular write process

Open
#3,604 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
2k
Forks
976
Avg merge
6d 15h
Merged PRs (30d)
7

Description

**BUG REPORT**

***Describe the bug***

we use SortedLedgerStorage+EntryLogManagerForSingleEntryLog.
When the entrylog file managed by EntryLogManagerForSingleEntryLog reaches the upper limit, the createNewLog method is executed. This operation is not locked, which will conflict with the addEntry executed in the compaction process, resulting in disordered entryLog file data.
![14e56f8a3e057508f446abb1ccd7b687](https://user-images.githubusercontent.com/35036009/199918343-ce3f5648-6851-4d82-b093-e0605ea9a02a.png)
![ae141da9c0fa6d51373775c19e0860ad](https://user-images.githubusercontent.com/35036009/199918357-d0124ed3-d7d7-4563-bef1-64ab62d553dc.png)
There is a concurrency problem with the two operations shown above.
In our case we found this code execution order:
1.logChannel.flush();
2.logChannel.write(sizeBuffer);
3.logChannel.appendLedgersMap();
4.logChannel.write(entry);

The correct entryLog file format should be like this:
header
entrySize1+ledgerId1+entryData1
entrySize2+ledgerId2+entryData2
...
entrySizen+ledgerIdn+entryDatan
ledgersMap

The wrong entrylog is as follows:
header
entrySize1+ledgerId1+entryData1
entrySize2+ledgerId2+entryData2
...
entrySizen
ledgersMap
ledgerIdn+entryDatan

I used the scanEntryLog method in EntryLogger to parse an error file. The phenomenon is as follows:
image
image

@hangc0276

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.