pingcap / pingcap/tidb

Backup txn kv data via br

Open
#38,506 7 comments 1 reaction 0 assignees View on GitHub
component/br type/feature-request
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

**Is your feature request related to a problem? Please describe:**

We can backup `raw` API data via `br` for `tikv` cluster. but for `txn` API data. we don't have such way to do so. and this will decrease the cluster operation and maintenance capabilities of data loss.

**Describe the feature you'd like:**

Backup `txn` API data via `br`.

**Describe alternatives you've considered:**

We can implement this feature in a simple way. we can just allow backup full range `txn` data first. the basic code logic is as below:
### Backup
- [ ] Add a new `txn` subcommand in [`backup`](https://github.com/pingcap/tidb/blob/master/br/cmd/br/backup.go#L154) just like `raw` command.
- [ ] Implement the `RunBackupTxn` function. just like [RunBackup](https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/backup.go#L277)
- [ ] Get global snapshot timestamp from `pd` and stop moving GC forward. just like [here](https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/backup.go#L341-L360)
- [ ] Build the backup key range. firstly we can simply transform a full backup range from utils. https://github.com/pingcap/tidb/blob/1659c3ea486d16ee161b28b7efad43e2f641f808/util/ranger/points.go#L168
- [ ] Reuse [backupRanges](https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/backup/client.go#L561) and the default backup request can be same as tidb backup
https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/backup.go#L381-L391
- [ ] (optional challenging work) abstract `RunBackup` and `RunBackupTxn`.

### Restore
- [ ] Add a new `txn` subcommand in [`restore`](https://github.com/pingcap/tidb/blob/master/br/cmd/br/restore.go#L177)
- [ ] Distinguish the tidb backup and `txn` backup after [read](https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/common.go#L646) `BackupMeta` file by condition
```
func isTxnBackup(backupMeta *pb.BackupMeta) bool {
return len(backupMeta.Schemas) == 0 && !backupMeta.IsRawKv
}
```
- [ ] Implement the `RunRestoreTxn` function. just like [RunRestoreRaw](https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/restore_raw.go#L61). because we don't need to create tables. so the most logic is as same as raw restore.
- [ ] Get backup file ranges by iterate backupMeta.Files.
- [ ] Split ranges to let region distribute evenly. https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/restore_raw.go#L149
- [ ] Restore backup files but ignore start key and end key https://github.com/pingcap/tidb/blob/aa49a4c072b8cec6d3f6b60cc1453fe60ae3dd48/br/pkg/task/restore_raw.go#L160

**Teachability, Documentation, Adoption, Migration Strategy:**

### Test
- [ ] Add an integration test, just like [`br_rawkv`](https://github.com/pingcap/tidb/tree/master/br/tests/br_rawkv).
- [ ] Add a go `txn` client. basically can write `txn` data to `TiKV` and check `txn` data is enough.
- [ ] Implement the test in `run.sh` and comment `/run-integration-br-tests` on a PR will trigger the test.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.