apache / apache/pulsar

for ungraceful shutdown ledgers , recoverry may cause data inconsostency.

Open
#15,782 1 comment 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**
When a broker is shutdowm suddenly, for example it holds ledger L1 and LAC is E0, has pending writes E1 and E2, WQ is B1, B2, B3. E1 has been written to B1, E2 has been written to B3. Then before bookkeeper client opens a new ledger and it is supposed to try to recover L1. There maybe some problem for current recovery policy. Recovery has two principles.
principle 1: One positive bookie will be judged as recoverable.
principle 2: (WQ-AQ)+1 negative bookie will be judged as unrecoverable.
positive bookie mean it has the entry, negative bookie mean it will respond NoSuchEntry when you query the entry.
For WQ=3 and only one bookie has the entry, it will be judged as recoverable if its response arrives earlier than the other two. It will be judged as unrecoverable and truncated if its response arrives later than the pther two. For the same condition is has two behavior.

Here is an example. when a broker is shutdowm suddenly, it holds ledger L1 and LAC is E0, has pending writes E1 and E2, WQ is B1, B2, B3. E1 has been written to B1, E2 has been written to B3. Then before bookkeeper client opens a new ledger and it is supposed to try to recover L1. Then it may have the following two cases.

case 1:

| time | B1 | B2. | B3 | is recoverable? |
| ------------- | ------------- |------------- | ------------- |------------- |
|t1. | OK | Waiting response. | Waiting response | Yes. |

case 2:
| time | B1 | B2. | B3 | is recoverable? |
| ------------- | ------------- |------------- | ------------- |------------- |
|t1. |NoSuchEntry | Waiting response. | Waiting response | Need more response|
|t2. | | NoSuchEntry. | Waiting response | No. |

case 1 and case 2 has different results. For case 1, client think the entry is not written success but bookie has the entry, this can cause inconsitency of clien and server.

**Expected behavior**
For the same condition, recovery should have the same results, not probably can recover, probably cann't recover.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the ledger recovery policy described in the issue, focusing on how responses from B1, B2, and B3 are handled when entries arrive in different orders. Reproduce both timing cases with WQ=3 and pending writes E1 and E2; done means the same recovery result is produced for the same ledger state regardless of response order.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.