TiKV StatefulSet is never created on first bootstrap when TidbCluster includes TiFlash

Bug Report

What version of Kubernetes are you using?

Reproduced in PingCAP test-infra (ksyun-rhel). Not specific to one Kubernetes version.

What version of TiDB Operator are you using?

The current v1 controller on master still has this path:

• pkg/controller/tidbcluster/tidb_cluster_control.go syncs TiFlash before TiKV
• pkg/manager/member/tiflash_member_manager.go treats GetStores failure as fatal and returns

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

Not relevant. TiKV StatefulSet is not created, so this is not a PVC/storage-class scheduling problem.

What's the status of the TiDB cluster pods?

PD is running. TiFlash StatefulSet may already exist. TiKV StatefulSet / pods are not created. The cluster never bootstraps.

What did you do?

Same shape as #4807, but this is a first-time create, not EBS restore:

1. Create a TidbCluster with PD + TiKV + TiFlash in the same spec (spec.recoveryMode is not set).
2. Optional but easier to hit: spec.pd.mode: ms with PD microservices (tso / scheduling).
3. Wait for the cluster to become Ready.

Concrete failing run: TCMS plan execution 8206262 (ng-import-into-br-migrate-tidbx, one TidbCluster with PD + 3 TiKV + 1 TiFlash write).

What did you expect to see?

Operator creates the TiKV StatefulSet, PD bootstraps, TidbCluster becomes Ready.

What did you see instead?

PD comes up, TiKV is never created, cluster stays not Ready until timeout.

Operator calls PD GET /pd/api/v1/stores from TiFlash sync and gets:

[PD:cluster:ErrNotBootstrapped]
TiKV cluster not bootstrapped, please start TiKV first

This is the same operator log as #4807:

Fail to GetStores for TidbCluster ...:
Error response 500 URL http://<pd>/pd/api/v1/stores,
body response: "[PD:cluster:ErrNotBootstrapped]TiKV cluster not bootstrapped, please start TiKV first"

After that, reconcile returns and tikvMemberManager.Sync never runs, so TiKV pods cannot appear.

This is not a resource-pool issue. In 8206262, sibling items (minio, tikv-worker, sdkserver, tools, ...) became Ready; only the TidbCluster items stayed not Ready until creationWaitTimeout.

Why this is an operator code bug

PD returning ErrNotBootstrapped before any TiKV exists is expected. The bug is that TiFlash sync aborts the whole reconcile on that error, so TiKV is never created, so PD can never bootstrap.

1. Reconcile order (tidb_cluster_control.go): tiflashMemberManager.Sync runs first; any error returns before tikvMemberManager.Sync.

2. After the TiFlash StatefulSet exists, syncTidbClusterStatus does:

storesInfo, err := pdCli.GetStores()
if err != nil {
    tc.Status.TiFlash.Synced = false
    klog.Warningf("Fail to GetStores for TidbCluster %s/%s: %s", ...)
    return err // blocks TiKV creation
}

3. TiKV already handles the same error and still creates the StatefulSet:

storesInfo, err := pdCli.GetStores()
if err != nil {
    if pdapi.IsTiKVNotBootstrappedError(err) {
        klog.Infof("TiKV of Cluster %s/%s not bootstrapped yet", ...)
        tc.Status.TiKV.BootStrapped = false
        return nil // continue and create STS
    }
    return err
}

Resulting deadlock:

1. PD becomes available.
2. TiFlash STS can be created (GetStores is skipped while the STS does not exist yet).
3. If TiKV STS is not created in that same loop (for example PD API is up but PDMS clients are not ready, so TiKV Sync requeues), the next loop sees an existing TiFlash STS, GetStores returns ErrNotBootstrapped, TiFlash returns error, TiKV Sync never runs again.

This is timing-dependent. If TiKV STS is created in the same reconcile as the first TiFlash STS, the cluster can come up. If the next reconcile hits GetStores first, TiKV is stuck forever.

Relation to #4807 / #4808

#4807 was the same Fail to GetStores / ErrNotBootstrapped blocking TiKV creation, during EBS snapshot restore.

#4808 only skipped TiFlash sync while spec.recoveryMode is true (syncRecoveryForTiFlash). It did not change:

• reconcile order (still TiFlash then TiKV)
• TiFlash GetStores still return err on ErrNotBootstrapped

So the first-bootstrap path is still broken.

Suggested fix

Either:

• Sync TiKV before TiFlash, or
• In TiFlash syncTidbClusterStatus, treat pdapi.IsTiKVNotBootstrappedError(err) the same way TiKV does (return nil) so later tikvMemberManager.Sync can create the StatefulSet.

Start in pkg/controller/tidbcluster/tidb_cluster_control.go and compare the TiFlash and TiKV sync paths in pkg/manager/member/tiflash_member_manager.go. Trace how pdCli.GetStores handles pdapi.IsTiKVNotBootstrappedError during first bootstrap. Done means a TidbCluster with PD, TiKV, and TiFlash creates the TiKV StatefulSet and can become Ready instead of stopping reconciliation.