for ungraceful shutdown ledgers , recoverry may cause data inconsostency.
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
When a broker is shutdowm suddenly, for example it holds ledger L1 and LAC is E0, has pending writes E1 and E2, WQ is B1, B2, B3. E1 has been written to B1, E2 has been written to B3. Then before bookkeeper client opens a new ledger and it is supposed to try to recover L1. There maybe some problem for current recovery policy. Recovery has two principles.
principle 1: One positive bookie will be judged as recoverable.
principle 2: (WQ-AQ)+1 negative bookie will be judged as unrecoverable.
positive bookie mean it has the entry, negative bookie mean it will respond NoSuchEntry when you query the entry.
For WQ=3 and only one bookie has the entry, it will be judged as recoverable if its response arrives earlier than the other two. It will be judged as unrecoverable and truncated if its response arrives later than the pther two. For the same condition is has two behavior.
Here is an example. when a broker is shutdowm suddenly, it holds ledger L1 and LAC is E0, has pending writes E1 and E2, WQ is B1, B2, B3. E1 has been written to B1, E2 has been written to B3. Then before bookkeeper client opens a new ledger and it is supposed to try to recover L1. Then it may have the following two cases.
case 1:
| time | B1 | B2. | B3 | is recoverable? |
| ------------- | ------------- |------------- | ------------- |------------- |
|t1. | OK | Waiting response. | Waiting response | Yes. |
case 2:
| time | B1 | B2. | B3 | is recoverable? |
| ------------- | ------------- |------------- | ------------- |------------- |
|t1. |NoSuchEntry | Waiting response. | Waiting response | Need more response|
|t2. | | NoSuchEntry. | Waiting response | No. |
case 1 and case 2 has different results. For case 1, client think the entry is not written success but bookie has the entry, this can cause inconsitency of clien and server.
**Expected behavior**
For the same condition, recovery should have the same results, not probably can recover, probably cann't recover.
Contributor guide
Research direction
Start by tracing the ledger recovery policy described in the issue, focusing on how responses from B1, B2, and B3 are handled when entries arrive in different orders. Reproduce both timing cases with WQ=3 and pending writes E1 and E2; done means the same recovery result is produced for the same ledger state regardless of response order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100