apache / apache/kvrocks

clusterx migrate support data synchronization mode

Open
#3,222 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
4.4k
Forks
658
Avg merge
1d 20h
Merged PRs (30d)
10

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/kvrocks/issues) and found no similar issues.

### Motivation

After the kvrocks migrate command completes the migration of existing and incremental data, it sets the migrated_slots_ variable and responds to subsequent write requests with the MOVED command, ensuring that subsequent requests are executed only on the migration target node.

In a scenario where we use the migration function to upgrade Kvrocks, we need to observe the data migration to the new kvrocks node for a period of time to ensure that the new node is functioning properly before returning the MOVED command to subsequent write requests. During this observation period, the incremental data synchronization operation is ongoing, meaning that the data is available on both the old and new primary nodes.

### Solution

This approach effectively implements phased verification of the new kvrocks program and enables rapid, lossless rollbacks.
To implement the above features, three subcommands need to be added to the clusterx migrate command:
clusterx migrate dtsmode: Data migration mode, continuously synchronizing incremental data
clusterx migrate cancel: Cancels the data migration
clusterx migrate dtscomplete: Completes the migration task, aligning the new primary with the old primary's offset

We will implement this feature. If you have better suggestions, please let us know.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the clusterx migrate command entry point and trace how migration completion sets the migrated state and handles subsequent writes. Review the existing migration flow before defining the three requested subcommands: dtsmode, cancel, and dtscomplete; done means incremental synchronization, cancellation, and offset alignment support the staged upgrade workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.