apache / apache/pulsar

Inconsistent Bookie ledgers and under replicate metadata in Bookie

Open
#14,573 3 comments 1 reaction 0 assignees View on GitHub
area/admin lifecycle/stale Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**

Check Bookie node log info, we can see the following error message:

```
20:15:20.948 [BookKeeperClientWorker-OrderedExecutor-24-0] ERROR org.apache.bookkeeper.replication.ReplicationWorker - Received error: -7 while trying to read entry: 19 of ledger: 6522968 in ReplicationWorker
```

Continue to track the log information of this ledger failure, we can see that in the past 30 days, the backup action of this ledger has been failing, as follows:

image

At this point we use the CLI command provided by bookie to check the status of the current ledger

```
bin/bookkeeper shell ledger -m 6522968
```

Output as follows:

image

We can see that there is no data information for this ledger in the bookie cluster

And check under replicate:

```
bin/bookkeeper shell listunderreplicated
```

Output as follows:

wecom-temp-214b6834b2b7930266f9eba438bec694

We can see that the information of this ledger no longer exists in the Bookie cluster, but the information of this ledger can still be found in the underreplicated list, so ReplicationWorker will always try to copy the information of this ledger to other Bookie nodes, But in fact this ledger no longer exists, so the replication behavior of ReplicationWorker will continue to fail.

Contributor guide

Open the contributing guide

Research direction

Start with the ReplicationWorker log path and reproduce the ledger state using `bin/bookkeeper shell ledger -m 6522968` and `bin/bookkeeper shell listunderreplicated`. Trace how a missing ledger remains in the underreplicated list; done means the stale entry is handled and ReplicationWorker no longer repeatedly fails trying to copy it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.