pingcap / pingcap/tidb

Flaky test: real-tikv SetupSuite fails with `cluster is not bootstrapped` (importintotest2 / importintotest / addindextest4)

Open
#70,483 1 comment 0 reactions 0 assignees View on GitHub
component/test flaky-test severity/minor type/bug
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Bug Report (flaky test / real-tikv CI setup)

### 1. Minimal reproduce step (Required)

Intermittent — no code trigger. Observed in the `idc-jenkins-ci-tidb/check_dev_2` real-tikv matrix.

- Run the real-tikv suites via `run_real_tikv_tests.sh` (e.g. `bazel_importintotest2`).
- Occasionally the suite's `SetupSuite` opens the store before the freshly started TiKV/PD cluster has finished bootstrapping.

Observed build: `idc-jenkins-ci-tidb/check_dev_2` #7220 (https://prow.tidb.net/jenkins/job/pingcap/job/tidb/job/ghpr_check2/7220/display/redirect), on PR #70153.

In that single run the same failure hit **three independent suites at once** — `importintotest2`, `importintotest`, and `addindextest4` — which is a shared-environment (cluster-bootstrap) symptom rather than a per-suite bug.

### 2. What did you expect to see? (Required)

The TiKV/PD cluster is fully bootstrapped before a real-tikv suite's `SetupSuite` opens the store, so `CreateMockStoreAndSetup` succeeds and the suite runs.

### 3. What did you see instead (Required)

`SetupSuite` fails in ~0.02s while opening the store, before any test logic runs:

```
--- FAIL: TestImportInto (0.02s)
testkit.go:284:
Error: Received unexpected error:
type:NOT_BOOTSTRAPPED message:"cluster is not bootstrapped"
```

The failure is entirely in store open / cluster bootstrap:

Stack trace

```
github.com/tikv/pd/client.gcStatesClient.GetGCState gc_client.go:324
github.com/tikv/client-go/v2/tikv.(*KVStore).loadTxnSafePoint tikv/kv.go:203
github.com/tikv/client-go/v2/tikv.NewKVStore tikv/kv.go:365
pkg/store/driver.(*TiKVDriver).OpenWithOptions tikv_driver.go:218
pkg/store/driver.(*TiKVDriver).Open tikv_driver.go:111
tests/realtikvtest.CreateMockStoreAndDomainAndSetup testkit.go:283
tests/realtikvtest.CreateMockStoreAndSetup testkit.go:225
tests/realtikvtest/importintotest2.(*mockGCSSuite).SetupSuite main_test.go:64
```

### 4. What is your TiDB version? (Required)

master (branch `master`); reproduced on the CI head of PR #70153.

---

### Analysis (evidence-backed)

- PD returning `NOT_BOOTSTRAPPED` means the cluster has no bootstrap info yet — i.e. no TiKV store has registered with PD when the test connects. This is cluster/environment state, not a code assertion.
- The failure path is fully inside vendored client-go / `pkg/store/driver` / the real-tikv test harness; it is reached from `SetupSuite`, before any suite logic executes.
- The same failure struck three unrelated suites in one run, consistent with the shared CI cluster not being bootstrapped in time, not a defect in any single suite.
- Related prior flaky reports for the same test family: #69772 (open), #69622 (closed) — different suite/subtest, different root cause (test logic), but confirms `TestImportInto` has a flaky history.

Suggested remedy: make the real-tikv harness wait for cluster bootstrap (retry `Open` on `NOT_BOOTSTRAPPED`) before starting suites, and `/retest` unblocks affected PRs in the meantime.

Suggested labels: component/test, flaky-test

Contributor guide

Open the contributing guide

Research direction

Run the real-tikv suites through run_real_tikv_tests.sh and inspect tests/realtikvtest/testkit.go, especially CreateMockStoreAndSetup and the SetupSuite entry points. Trace the store-open path around the NOT_BOOTSTRAPPED failure and verify that importintotest2, importintotest, and addindextest4 start only after cluster bootstrap and complete successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, distributed-systems, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.