apache / apache/incubator-pegasus

BUG(split): When a child copies mutations, the order of these mutations may be inconsistent.

Open
#2,210 0 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Bug Report

1. What did you do?
There are two places that may cause this issus.
The first is copying prepare_list.
1. copy prepare_list
1. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L305-L308
2. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/prepare_list.cpp#L52-L57
3. During the process of copying the prepare_list, write operations were not stopped, so mutation_cache and last_commit_decree will continue to grow.
4. Since last_commit_decree is copied later, it is possible for last_committed_decree to be greater than the actual committed_decree in the prepare_list.
5. Therefore, this may lead to data loss.
2. plog append
1. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L370-L378
2. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L1359-L1366
3. Currently, when the child replicates mutations, there are also asynchronous writes coming from the client, and both operations will perform plog append.
4. The plog does not have a method to distinguish the order.
2. soulution
1. For the first issue, it can be resolved by copying the mutation_cache again after copying the last_commited_decree.
2. For the second issue, there is currently no good solution in mind.
3. Stopping client writes before the copy_prepare_list process seems like a good approach to prevent conflicts and ensure consistency.
4. By stopping client writes before the copy_prepare_list process, you can address both of the issues.
3. Here is the test for the write-stop duration

数据量 1M
| 分片数量 | 单分片阻写时间(min) | 单分片阻写时间(max) | 单分片阻写时间(avg) | 总体阻写时间 |
|--------------|---------------------|---------------------|---------------------|--------------|
| 32->64 | 1.1s | 2.1s | 1.9s | 85s |
| 64->128 | 1.5s | 61s | 4.2s | 83s |
| 128->256 | 1.3s | 5.9s | 3.2s | 85s |
| 256->512 | 1.3s | 32s | 3.1s | 89s |
| 512->1024 | 1.3s | 60s | 7.1s | 110s |

数据量 172G
| 分片数量 | 单分片阻写时间(min) | 单分片阻写时间(max) | 单分片阻写时间(avg) | 总体阻写时间 |
|--------------|---------------------|---------------------|---------------------|--------------|
| 64->128 | 1.2s | 121.5s | 20.9s | 141s |
| 128->256 | 1.8s | 81.5s | 14.9s | 122s |
| 256->512 | 2s | 66.1s | 11s | 144s |
| 512->1024 | (内存被打满 OOM) | | | |

Contributor guide

Open the contributing guide

Research direction

Start with the referenced split logic in src/replica/split/replica_split_manager.cpp, especially the prepare_list copy and plog append sections, then read src/replica/prepare_list.cpp. Reproduce or reason through concurrent child replication and client writes; done means mutation order and commit state remain consistent during splitting without data loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.