apache / apache/incubator-pegasus

The backup-cluster may doing an incomplete learn with duplication

Open
#2,107 1 comment 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
C++
Stars
2.1k
Forks
328
PR merge metrics
No merged PRs in 30d

Description

## Bug Report

At present, the implementation of dup is that when the backup-cluster executes the dup rpc processing function, multiple requests in dup are written to rocksdb in multiple times.
Each time it is written to rocksdb, the decree of the dup mutation is written at the same time. If the backup-cluster is checkpointed at this time, the data of the decree may not be completely written to rocksdb.
If the learner of the backup-cluster uses this checkpoint to start learning, it will start to request plog from decree+1 after learning. As a result, some dup requests of the decree are not learned, and some data is lost.

```
int pegasus_write_service::duplicate(int64_t decree,
const dsn::apps::duplicate_request &requests,
dsn::apps::duplicate_response &resp)
{
// If the `for` loop has not yet been completed, and there is a need to checkpoint.
// The checkpoint may not include all data cause these request share the same decree.
// In other word, this creates an inconsistency.
for (const auto &request : requests.entries) {
// ...
}
}
```

Contributor guide

Open the contributing guide

Research direction

Start at pegasus_write_service::duplicate, especially the loop over requests.entries, and trace how checkpointing and backup-cluster learning handle a shared decree. Done means a checkpoint cannot cause the learner to skip duplicate requests from that decree or lose data; the issue does not name a file or test to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.