apache / apache/incubator-pegasus
The backup-cluster may doing an incomplete learn with duplication
- Dominant language
- C++
- Stars
- 2.1k
- Forks
- 328
- PR merge metrics
- No merged PRs in 30d
Description
## Bug Report
At present, the implementation of dup is that when the backup-cluster executes the dup rpc processing function, multiple requests in dup are written to rocksdb in multiple times.
Each time it is written to rocksdb, the decree of the dup mutation is written at the same time. If the backup-cluster is checkpointed at this time, the data of the decree may not be completely written to rocksdb.
If the learner of the backup-cluster uses this checkpoint to start learning, it will start to request plog from decree+1 after learning. As a result, some dup requests of the decree are not learned, and some data is lost.
```
int pegasus_write_service::duplicate(int64_t decree,
const dsn::apps::duplicate_request &requests,
dsn::apps::duplicate_response &resp)
{
// If the `for` loop has not yet been completed, and there is a need to checkpoint.
// The checkpoint may not include all data cause these request share the same decree.
// In other word, this creates an inconsistency.
for (const auto &request : requests.entries) {
// ...
}
}
```
Contributor guide
Research direction
Start at pegasus_write_service::duplicate, especially the loop over requests.entries, and trace how checkpointing and backup-cluster learning handle a shared decree. Done means a checkpoint cannot cause the learner to skip duplicate requests from that decree or lose data; the issue does not name a file or test to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100