Inconsistent Bookie ledgers and under replicate metadata in Bookie
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
Check Bookie node log info, we can see the following error message:
```
20:15:20.948 [BookKeeperClientWorker-OrderedExecutor-24-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 19 of ledger: 6522968 in ReplicationWorker
```
Continue to track the log information of this ledger failure, we can see that in the past 30 days, the backup action of this ledger has been failing, as follows:

At this point we use the CLI command provided by bookie to check the status of the current ledger
```
bin/bookkeeper shell ledger -m 6522968
```
Output as follows:

We can see that there is no data information for this ledger in the bookie cluster
And check under replicate:
```
bin/bookkeeper shell listunderreplicated
```
Output as follows:

We can see that the information of this ledger no longer exists in the Bookie cluster, but the information of this ledger can still be found in the underreplicated list, so ReplicationWorker will always try to copy the information of this ledger to other Bookie nodes, But in fact this ledger no longer exists, so the replication behavior of ReplicationWorker will continue to fail.
Contributor guide
Research direction
Start with the ReplicationWorker log path and reproduce the ledger state using `bin/bookkeeper shell ledger -m 6522968` and `bin/bookkeeper shell listunderreplicated`. Trace how a missing ledger remains in the underreplicated list; done means the stale entry is handled and ReplicationWorker no longer repeatedly fails trying to copy it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100