Flaky test: real-tikv SetupSuite fails with `cluster is not bootstrapped` (importintotest2 / importintotest / addindextest4)
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Bug Report (flaky test / real-tikv CI setup)
### 1. Minimal reproduce step (Required)
Intermittent — no code trigger. Observed in the `idc-jenkins-ci-tidb/check_dev_2` real-tikv matrix.
- Run the real-tikv suites via `run_real_tikv_tests.sh` (e.g. `bazel_importintotest2`).
- Occasionally the suite's `SetupSuite` opens the store before the freshly started TiKV/PD cluster has finished bootstrapping.
Observed build: `idc-jenkins-ci-tidb/check_dev_2` #7220 (https://prow.tidb.net/jenkins/job/pingcap/job/tidb/job/ghpr_check2/7220/display/redirect), on PR #70153.
In that single run the same failure hit **three independent suites at once** — `importintotest2`, `importintotest`, and `addindextest4` — which is a shared-environment (cluster-bootstrap) symptom rather than a per-suite bug.
### 2. What did you expect to see? (Required)
The TiKV/PD cluster is fully bootstrapped before a real-tikv suite's `SetupSuite` opens the store, so `CreateMockStoreAndSetup` succeeds and the suite runs.
### 3. What did you see instead (Required)
`SetupSuite` fails in ~0.02s while opening the store, before any test logic runs:
```
--- FAIL: TestImportInto (0.02s)
testkit.go:284:
Error: Received unexpected error:
type:NOT_BOOTSTRAPPED message:"cluster is not bootstrapped"
```
The failure is entirely in store open / cluster bootstrap:
Stack trace
```
github.com/tikv/pd/client.gcStatesClient.GetGCState gc_client.go:324
github.com/tikv/client-go/v2/tikv.(*KVStore).loadTxnSafePoint tikv/kv.go:203
github.com/tikv/client-go/v2/tikv.NewKVStore tikv/kv.go:365
pkg/store/driver.(*TiKVDriver).OpenWithOptions tikv_driver.go:218
pkg/store/driver.(*TiKVDriver).Open tikv_driver.go:111
tests/realtikvtest.CreateMockStoreAndDomainAndSetup testkit.go:283
tests/realtikvtest.CreateMockStoreAndSetup testkit.go:225
tests/realtikvtest/importintotest2.(*mockGCSSuite).SetupSuite main_test.go:64
```
### 4. What is your TiDB version? (Required)
master (branch `master`); reproduced on the CI head of PR #70153.
---
### Analysis (evidence-backed)
- PD returning `NOT_BOOTSTRAPPED` means the cluster has no bootstrap info yet — i.e. no TiKV store has registered with PD when the test connects. This is cluster/environment state, not a code assertion.
- The failure path is fully inside vendored client-go / `pkg/store/driver` / the real-tikv test harness; it is reached from `SetupSuite`, before any suite logic executes.
- The same failure struck three unrelated suites in one run, consistent with the shared CI cluster not being bootstrapped in time, not a defect in any single suite.
- Related prior flaky reports for the same test family: #69772 (open), #69622 (closed) — different suite/subtest, different root cause (test logic), but confirms `TestImportInto` has a flaky history.
Suggested remedy: make the real-tikv harness wait for cluster bootstrap (retry `Open` on `NOT_BOOTSTRAPPED`) before starting suites, and `/retest` unblocks affected PRs in the meantime.
Suggested labels: component/test, flaky-test
Contributor guide
Research direction
Run the real-tikv suites through run_real_tikv_tests.sh and inspect tests/realtikvtest/testkit.go, especially CreateMockStoreAndSetup and the SetupSuite entry points. Trace the store-open path around the NOT_BOOTSTRAPPED failure and verify that importintotest2, importintotest, and addindextest4 start only after cluster bootstrap and complete successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100