TiDB operator cannot scale up the tiflash after scaling down to 0

Bug Report

What version of Kubernetes are you using?
Client Version: v1.31.1
Kustomize Version: v5.4.2

What version of TiDB Operator are you using?
v1.6.0

What's the status of the TiDB cluster pods?
TiFlash pods are in CrashBackOffLoop State.

What did you do?
We scaled in tiflash from 3 to 0 and then scaled it out from 0 to 3.

How to reproduce

1. Deploy a TiDB cluster with tiflash enabled, for example:

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: test-cluster
spec:
  configUpdateStrategy: RollingUpdate
  enableDynamicConfiguration: true
  helper:
    image: alpine:3.16.0
  pd:
    baseImage: pingcap/pd
    # config: "[dashboard]\n  internal-proxy = true\n"
    maxFailoverCount: 0
    mountClusterClientSecret: true
    replicas: 3
    requests:
      storage: 10Gi
  pvReclaimPolicy: Retain
  tidb:
    baseImage: pingcap/tidb
    config: "[performance]\n  tcp-keep-alive = true\ngraceful-wait-before-shutdown\
      \ = 30\n"
    maxFailoverCount: 0
    replicas: 3
    service:
      externalTrafficPolicy: Local
      type: NodePort
  tikv:
    baseImage: pingcap/tikv
    config: |
      [raftdb]
        max-open-files = 256
      [rocksdb]
        max-open-files = 128
    maxFailoverCount: 0
    mountClusterClientSecret: true
    replicas: 3
    requests:
      storage: 100Gi
  tiflash:
    baseImage: pingcap/tiflash
    version: "v8.1.0"
    replicas: 3
    config:
      # configure the configuration file for TiFlash process
      config: |
        [logger]
          log = "/data0/logs/somelog"
      proxy: |
        [security]
          cert-allowed-cn = ["CNNAME"]
    storageClaims:
      - resources:
          requests:
            storage: 10Gi
  timezone: UTC
  version: v8.1.0

2. Change the spec.tiflash.replicas from 3 to 0:
3. After the operator finishes the reconciliation, change the spec.tiflash.replicas back to 3.

What did you expect to see?
We expected that TiFlash pods are running and be in Healthy state

What did you see instead?
The Tiflash pods kept crashing and be in CrashBackOffLoop state.

Root Cause
We think the root cause of this problem is that when scaling in the TiFlash, the stores will be in Tombstone state. After we change the spec.tiflash.replicas from 0 to 3, the operator will delete the original statefulset and create a new one with replicas set to 3 instead of changing the original statefulset. This behaviour bypasses the ScaleOut function at this line https://github.com/pingcap/tidb-operator/blob/master/pkg/manager/member/tiflash_scaler.go#L52.
After encounter this issue, the user cannot simply delete the CR and apply it again to make the TiFlash run correctly as the operator will not delete pvcs after user deletes CR causing the new cluster reusing the stores that in Tombstone state.

Start in pkg/manager/member/tiflash_scaler.go around line 52, especially the ScaleOut path, and reproduce the reported scale from 3 to 0 and back to 3 using the provided TidbCluster configuration. Trace how the operator deletes and recreates the StatefulSet and how TiFlash stores reach Tombstone state. Done means scaling back to 3 restores running TiFlash pods in Healthy state without reusing stores incorrectly.