apache / apache/incubator-pegasus
BUG(split): When a child copies mutations, the order of these mutations may be inconsistent.
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Bug Report
1. What did you do?
There are two places that may cause this issus.
The first is copying prepare_list.
1. copy prepare_list
1. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L305-L308
2. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/prepare_list.cpp#L52-L57
3. During the process of copying the prepare_list, write operations were not stopped, so mutation_cache and last_commit_decree will continue to grow.
4. Since last_commit_decree is copied later, it is possible for last_committed_decree to be greater than the actual committed_decree in the prepare_list.
5. Therefore, this may lead to data loss.
2. plog append
1. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L370-L378
2. https://github.com/apache/incubator-pegasus/blob/a875cb4261730a5eb5408bb79688ff447cb47135/src/replica/split/replica_split_manager.cpp#L1359-L1366
3. Currently, when the child replicates mutations, there are also asynchronous writes coming from the client, and both operations will perform plog append.
4. The plog does not have a method to distinguish the order.
2. soulution
1. For the first issue, it can be resolved by copying the mutation_cache again after copying the last_commited_decree.
2. For the second issue, there is currently no good solution in mind.
3. Stopping client writes before the copy_prepare_list process seems like a good approach to prevent conflicts and ensure consistency.
4. By stopping client writes before the copy_prepare_list process, you can address both of the issues.
3. Here is the test for the write-stop duration
数据量 1M
| 分片数量 | 单分片阻写时间(min) | 单分片阻写时间(max) | 单分片阻写时间(avg) | 总体阻写时间 |
|--------------|---------------------|---------------------|---------------------|--------------|
| 32->64 | 1.1s | 2.1s | 1.9s | 85s |
| 64->128 | 1.5s | 61s | 4.2s | 83s |
| 128->256 | 1.3s | 5.9s | 3.2s | 85s |
| 256->512 | 1.3s | 32s | 3.1s | 89s |
| 512->1024 | 1.3s | 60s | 7.1s | 110s |
数据量 172G
| 分片数量 | 单分片阻写时间(min) | 单分片阻写时间(max) | 单分片阻写时间(avg) | 总体阻写时间 |
|--------------|---------------------|---------------------|---------------------|--------------|
| 64->128 | 1.2s | 121.5s | 20.9s | 141s |
| 128->256 | 1.8s | 81.5s | 14.9s | 122s |
| 256->512 | 2s | 66.1s | 11s | 144s |
| 512->1024 | (内存被打满 OOM) | | | |
Contributor guide
Research direction
Start with the referenced split logic in src/replica/split/replica_split_manager.cpp, especially the prepare_list copy and plog append sections, then read src/replica/prepare_list.cpp. Reproduce or reason through concurrent child replication and client writes; done means mutation order and commit state remain consistent during splitting without data loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100