TiDB operator cannot update configuration for TiKV

Bug Report

What version of Kubernetes are you using?

1.28.1

What version of TiDB Operator are you using?

1.6.0

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

standard

What's the status of the TiDB cluster pods?

What did you do?

We tried to change the storage.engine option of TiKV, via the spec.tikv.config property. But we found that after changing storage.engine from raft-kv to partitioned-raft-kv, the TiKV keeps using raft-kv as the storage engine.

1. Install TiDB cluster by applying the CR.

apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
  version: v8.1.0
  timezone: UTC
  pvReclaimPolicy: Retain
  enableDynamicConfiguration: true
  configUpdateStrategy: RollingUpdate
  discovery: {}
  helper:
    image: alpine:3.16.0
  pd:
    baseImage: pingcap/pd
    maxFailoverCount: 0
    replicas: 1
    requests:
      storage: "1Gi"
    config: {}
  tikv:
    baseImage: pingcap/tikv
    maxFailoverCount: 0
    evictLeaderTimeout: 1m
    replicas: 1
    requests:
      storage: "1Gi"
    config:
      storage:
        # In basic examples, we set this to avoid using too much storage.
        reserve-space: "0MB"
        engine: "raft-kv"
      rocksdb:
        # In basic examples, we set this to avoid the following error in some Kubernetes clusters:
        # "the maximum number of open file descriptors is too small, got 1024, expect greater or equal to 82920"
        max-open-files: 256
      raftdb:
        max-open-files: 256
  tidb:
    baseImage: pingcap/tidb
    maxFailoverCount: 0
    replicas: 1
    service:
      type: ClusterIP
    config: {}

2. Change the spec.tikv.config.storage.engine to partitioned-raft-kv
3. Use mysqlsh to query the current configuration, and still see the engine is raft-kv

+------+----------------------------------------------------------------+----------------+---------+
| Type | Instance                                                       | Name           | Value    |
+------+----------------------------------------------------------------+----------------+---------+
| tikv | advanced-tidb-tikv-2.advanced-tidb-tikv-peer.default.svc:20160 | storage.engine | raft-kv |
| tikv | advanced-tidb-tikv-1.advanced-tidb-tikv-peer.default.svc:20160 | storage.engine | raft-kv |
| tikv | advanced-tidb-tikv-0.advanced-tidb-tikv-peer.default.svc:20160 | storage.engine | raft-kv |
+------+----------------------------------------------------------------+----------------+---------+

What did you expect to see?
Storage engine should be updated, or the config change should be rejected explicitly.

What did you see instead?
Although the Pods are rolling restarted with the updated ConfigMap, the actual configuration being used by the TiKV is not updated. We suspect that the updated configuration gets lost when the TiDB merges the configuration.

Reproduce the issue by applying the TiDB cluster CR, changing spec.tikv.config.storage.engine, and querying the running value with mysqlsh. Inspect the updated TiKV ConfigMap and the configuration merge path; done means the engine changes to partitioned-raft-kv or the change is explicitly rejected.