apache / apache/incubator-pegasus
BUG(split): When a table is performing a split, it may become stuck
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Bug Report
1. What did you do?
Two issues.
- one:
1. The primary receives the split information from the meta, executes the split asynchronously, and synchronizes the split information to the secondary via group_check.
2. Upon receiving the split information, the secondary shard starts executing the split.
3. The primary must receive notifications from all three child shards confirming that they have completed replicating all mutations before converting asynchronous mutation replication to synchronous replication and proceeding to the next phase.
4. However, if:
a. The primary receives the split request and begins executing the split asynchronously, while the secondary shards also start executing the split.
b. Suppose one of the secondaries completes replication and notifies the parent primary.
c. However, after receiving the notification, the parent primary encounters an issue with its own split and needs to redo it.
d. **The parent primary will clear all state information, including the previously received notifications.
e. The parent primary then starts a new round of splitting, but the previously completed child will not re-execute the process. As a result, the parent primary will never receive notifications from all three child shards in this new split process, causing it to wait indefinitely.**
- two:
The variable caught_up_children has a thread conflict issue.
2. Solution
For the first , **it only needs to ensure that during the split process, the primary child is the first to insert "caught_up_children"**.
For the secondary , the variable caught_up_children needs to be locked.
Contributor guide
Research direction
Start by tracing the primary and secondary split paths, including group_check, and locate the caught_up_children state and its concurrent accesses. Verify how a primary split retry clears notifications and how secondary updates race; done means retries do not wait indefinitely and caught_up_children is synchronized without losing valid child completion notifications.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100