BP-56: Support data migration
- Dominant language
- Java
- Stars
- 2k
- Forks
- 976
- Avg merge
- 6d 15h
- Merged PRs (30d)
- 7
Description
### Motivation
**Currently bookkeeper does not support data migration, only data recovery.**
We have a scenario where 125 bookie nodes are offline, and we find that the bookie's offline solution is very time-consuming.
Bookie offline steps are as follows:
1. Set the bookie node to be offline to readOnly;
2. Wait for the Pulsar data on the Bookie node to be offline to expire and delete;
3. When most of the data on these offline nodes is expired and cleaned up, there will still be some data that cannot be expired and deleted;
4. Stop a bookie, and then use the decommission command to migrate the data that has not been expired and cleaned up to the new node:
`bin/bookkeeper shell decommissionbookie -bookieid xx`
5. When the data on one bookie node is migrated, continue to the next bookie node;
Step 4 is very time-consuming. We found that waiting for a bookie data migration to complete, it takes about 1 hour, and we have 125 bookie nodes to be offline.
In addition, step 2 is also very time-consuming, depending on the pulsar retain time, usually more than ten hours.
### Proposal
To solve this problem, we developed a data migration tool.
After having this tool, our offline steps are as follows:
1. Execute the data migration command:
`bin/bookkeeper shell replicasMigration --bookieIds bookie1,bookie2 --readOnly true`
2. When the data migration is completed, stop all bookie nodes to be offline;
**The replicasMigration command works as follows:**
1. List the ledgers on the bookie nodes (bookie1 and bookie2) to be offline, and write them to zookeeper, the path organization on zookeeper is as follows:
> `ledgers/replicasMigration/ledgerId1`
And write the bookie nodes of the migrating replicas to the migration path, for example:
> `set ledgers/replicasMigration/ledgerId1 bookie1,bookie2`
2. Start the replica migration service ReplicasMigrationWorker in AutoRecoveryMain:
> The ReplicasMigrationWorker service first obtains a migrating ledger , then finds the fragments stored by the ledger on the corresponding bookie node, and replicates these fragments by the replicateLedgerFragment method.
3. When a ledger migration task is completed, the corresponding ledger path on zookeeper will be deleted, for example:
> `delete ledgers/replicasMigration/ledgerId1`
In addition, this command can also migrate some replicas data on bookie nodes to other nodes, for example:
`bin/bookkeeper shell replicasMigration --bookieIds bookie1,bookie2 --ledgerIds ledger1,ledger2,ledger3 --readOnly false`
### For example
1. Migrate all ledger data on bookie1 and bookie2 to other bookie nodes:
`sh bin/bookkeeper shell replicasMigration -bookieIds bookie1,bookie2 -readOnly true
`
2. Migrate ledger1 and ledger3 on bookie1 and bookie2 to other bookie nodes:
`sh bin/bookkeeper shell replicasMigration -bookieIds bookie1,bookie2 -ledgerIds ledger1,ledger3 -readOnly false`
### Application scenarios:
**1. The bookie node goes offline:**
As mentioned above, after bookkeeper has this data migration tool, the offline steps of bookie are only two steps, and the time-consuming is greatly reduced:
a.Execute the data migration command:
bin/bookkeeper shell replicasMigration --bookieIds bookie1,bookie2 --readOnly true
b. When the data migration is completed, stop all bookie nodes to be offline;
**2. Expand the bookie node to improve the reading speed of historical data:**
a. When the client consumes historical data a few days ago, it hopes to increase the reading speed of historical data by expanding the bookie node.
b. When we expand the new bookie node to the cluster, the new node can only receive the read and write of new data, and cannot improve the reading speed of historical data.
c. After the data migration tool of bookkeeper, we can migrate some historical data to the new node, let the new node provide some historical data reading, and improve the reading speed of historical data.
Contributor guide
Assessment
This issue has not been assessed yet.