pingcap / pingcap/tidb

BR flaky `br_region_rule` checksum mismatch in `pull_br_integration_test`

Open
#67,339 0 comments 0 reactions 0 assignees View on GitHub
contribution type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Summary

`br_region_rule` is flaky on `release-8.5` in `pull_br_integration_test`.

Confirmed failed builds with the same issue:

- [2099](https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftidb%2Frelease-8.5%2Fpull_br_integration_test/detail/pull_br_integration_test/2099/pipeline/)
- [2118](https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftidb%2Frelease-8.5%2Fpull_br_integration_test/detail/pull_br_integration_test/2118/pipeline/)
- [2188](https://do.pingcap.net/jenkins/blue/organizations/jenkins/pingcap%2Ftidb%2Frelease-8.5%2Fpull_br_integration_test/detail/pull_br_integration_test/2188/pipeline/)

This looks like a test flake rather than a PR-specific regression.

## Failure symptom

The failure is a restore checksum mismatch in `br_region_rule`, for example:

```text
checksum mismatch for table 'br_region_rule.sbtest103'
```

In the failed runs, expected data exists in backup metadata, but the restored table ends up empty, with signals like:

- `actual totalKvs=0`
- `actual totalBytes=0`

## Suspected root cause

The flaky point appears to be the checkpoint retry path in `br/tests/br_region_rule/run.sh`.

The test injects a restore failure after partial restore progress may already be persisted into checkpoint. A later retry reuses that checkpoint, but the target schema has been dropped first. This can make BR skip ranges that checkpoint thinks are already restored even though the actual restored data is gone, and the final checksum then fails.

## Suggested fix

Adjust `br_region_rule` so checkpoint retry matches checkpoint semantics:

- do not drop the target schema before retrying with the same checkpoint, or
- if schema is dropped, clear the related checkpoint and do a fresh restore

## Relevant files

- [br/tests/br_region_rule/run.sh](/Users/andy/Work/PingCAP/tidb/br/tests/br_region_rule/run.sh)
- [br/pkg/restore/snap_client/tikv_sender.go](/Users/andy/Work/PingCAP/tidb/br/pkg/restore/snap_client/tikv_sender.go)
- [br/pkg/restore/restorer.go](/Users/andy/Work/PingCAP/tidb/br/pkg/restore/restorer.go)
- [br/pkg/restore/snap_client/client.go](/Users/andy/Work/PingCAP/tidb/br/pkg/restore/snap_client/client.go)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.