pingcap / pingcap/tidb

v6.5.3 br full restore to downstream hit checksum mismatch, return succeeded

Open
#46,105 1 comment 0 reactions 1 assignee Claimed by @YuJuncen View on GitHub
component/br severity/moderate type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)
1. deploy two tidbcluster, one as ticdc upstream, the other is ticdc downstream.
2. Stop changefeed that sync from upstream to downstream.
3. backup in upstream.
4. drop database in downstream.
5. restore full backup in step 3 to downstream.

```
[2023/08/14 21:24:27.613 +08:00] [INFO] [ticdc.go:190] ["will run"] [cmd="tiup cdc:v6.5.3 cli changefeed remove --server=http://ticdc2-peer.e2e-dr-auto-sync-with-tools-tps-1891739-1-775:8300 --changefeed-id=dr-autosyn-changefeed "]
2023-08-14T21:24:27.616+0800 INFO k8s/client.go:132 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/08/14 21:24:28.247 +08:00] [INFO] [ticdc.go:199] ["remove successfully"] [output="Changefeed remove successfully.\nID: dr-autosyn-changefeed\nCheckpointTs: 443552081611849771\nSinkURI: mysql://root:xxxxx@cdc-downstream-tc-tidb.e2e-dr-auto-sync-with-tools-tps-1891739-1-775:4000/"]
[2023/08/14 21:24:28.247 +08:00] [INFO] [util.go:68] ["wait to start a new changefeed"] [wait=30s] [callBy=ticdc.go:200]
[2023/08/14 21:24:58.247 +08:00] [INFO] [util.go:60] ["sleep timeout"]
2023-08-14T21:24:58.252+0800 INFO k8s/client.go:132 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
2023-08-14T21:24:58.615+0800 INFO k8s/client.go:132 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/08/14 21:24:58.957 +08:00] [INFO] [ticdc.go:146] ["will run"] [cmd="tiup cdc:v6.5.3 cli changefeed list --server=http://ticdc2-peer.e2e-dr-auto-sync-with-tools-tps-1891739-1-775:8300"]
2023-08-14T21:24:58.961+0800 INFO k8s/client.go:132 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[]
2023-08-14T21:24:59.198+0800 INFO k8s/client.go:132 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[]
[2023/08/14 21:24:59.445 +08:00] [INFO] [ticdc.go:180] ["no changefeed in this cluster"]
2023-08-14T21:24:59.448+0800 INFO k8s/client.go:132 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/08/14 21:24:59.804 +08:00] [INFO] [ticdc.go:262] ["run backup full"] [brCmd="tiup br:v6.5.3 backup full --pd \"pd4-peer:2379\" --storage \"s3://tmp/ticdc-upstream-full-backup-2023-08-14T21:24:59+08:00?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3A%2F%2Fminio.pingcap.net%3A9000&force-path-style=true\" --ratelimit 60"]
2023-08-14T21:24:59.807+0800 INFO k8s/client.go:132 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
[2023/08/14 21:36:14.573 +08:00] [INFO] [cmd.go:160] ["Stop remote command since context is done"] [cmd="go-tpc tpcc run -D tpcc --host tiup-peer.e2e-dr-auto-sync-with-tools-tps-1891739-1-775 -P3390 --warehouses 1000 -T 50 --time 36000m --ignore-error '2013,1213,1105,1205,8022,8028,9004,9007,1062' --user root --password '' --interval '10s'"] [nodename=benchtoolset]
[2023/08/14 21:36:14.573 +08:00] [WARN] [cmd.go:165] ["Stop remote command failed"] [error="io: read/write on closed pipe"]
[2023/08/14 21:37:20.676 +08:00] [INFO] [ticdc.go:272] ["backup succeeded."] [output="[2023/08/14 21:37:20.632 +08:00] [INFO] [collector.go:73] [\"Full Backup success summary\"] [total-ranges=2415] [ranges-succeed=2415] [ranges-failed=0] [backup-checksum=2m19.425301897s] [backup-fast-checksum=6.449786ms] [backup-total-ranges=108] [total-take=12m20.704045157s] [BackupTS=443552760063852549] [total-kv=1128657293] [total-kv-size=127.5GB] [average-speed=172.1MB/s] [backup-data-size(after-compressed)=49.09GB] [Size=49087688442]"]
[2023/08/14 21:37:20.676 +08:00] [INFO] [ticdc.go:278] ["drop user db before restore full clsuter"]
[2023/08/14 21:37:20.694 +08:00] [INFO] [util.go:73] ["drop database PERFORMANCE_SCHEMA"]
[2023/08/14 21:37:20.694 +08:00] [INFO] [util.go:73] ["drop database METRICS_SCHEMA"]
[2023/08/14 21:37:20.695 +08:00] [INFO] [util.go:73] ["drop database tpcc"]
[2023/08/14 21:37:21.205 +08:00] [INFO] [util.go:73] ["drop database INFORMATION_SCHEMA"]
[2023/08/14 21:37:21.205 +08:00] [INFO] [util.go:73] ["drop database bank"]
[2023/08/14 21:37:21.715 +08:00] [INFO] [ticdc.go:287] ["run restore full"] [brCmd="tiup br:v6.5.3 restore full --pd \"cdc-downstream-tc-pd:2379\" --storage \"s3://tmp/ticdc-upstream-full-backup-2023-08-14T21:24:59+08:00?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http%3A%2F%2Fminio.pingcap.net%3A9000&force-path-style=true\" --send-credentials-to-tikv=true --check-requirements=false"]
2023-08-14T21:37:21.720+0800 INFO k8s/client.go:132 it should be noted that a long-running command will not be interrupted even the use case has ended. For more information, please refer to https://github.com/pingcap/test-infra/discussions/129
2023-08-14T21:42:05.092+0800 ERROR k8s/workloadnode.go:45 Exec error{error 26 0 command terminated with exit code 1}
github.com/pingcap/test-infra/sdk/resource/impl/k8s.(*WorkloadNode).Exec
/go/pkg/mod/github.com/pingcap/test-infra/sdk@v0.0.0-20230722082756-fec8c84e789f/resource/impl/k8s/workloadnode.go:45
github.com/pingcap/endless/pkg/workloadnode/tiup.(*workloadHost).Exec.func1
/home/jenkins/agent/workspace/endless-master-build/pkg/workloadnode/tiup/host.go:25
[2023/08/14 21:42:05.094 +08:00] [INFO] [cases.go:208] ["common test finished"]
```

### 2. What did you expect to see? (Required)

### 3. What did you see instead (Required)
Restore in step 6 failed due to check sum mismatch for database tpcc.
```
[2023/08/14 21:37:54.393 +08:00] [ERROR] [client.go:1375] ["failed in validate checksum"] [db=tpcc] [table=warehouse] ["origin tidb crc64"=15874918267568833621] ["calculated crc64"=10689267397864432177] ["origin tidb total kvs"=1000] ["calculated total kvs"=1000] ["origin tidb total bytes"=125065] ["calculated total bytes"=125065] [stack="[github.com/pingcap/tidb/br/pkg/restore.(*Client).GoValidateChecksum.func2.2\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1375\ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75](http://github.com/pingcap/tidb/br/pkg/restore.(*Client).GoValidateChecksum.func2.2/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/client.go:1375/ngithub.com/pingcap/tidb/br/pkg/utils.(*WorkerPool).ApplyOnErrorGroup.func1/n/t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/worker.go:76/ngolang.org/x/sync/errgroup.(*Group).Go.func1/n/t/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75)"]
```

### 4. What is your TiDB version? (Required)
6.5.3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.