pingcap / pingcap/tidb-operator

Fail to resize Tikv PVCs

Open
#6,035 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Bug Report

What version of Kubernetes are you using?

v1.26.15-aliyun.1 running on Alicloud ACK

What version of TiDB Operator are you using?

TiDB Operator Version: version.Info{GitVersion:"v1.4.5", GitCommit:"e98d820d5421b028015c341e3fa2db2e4016dbb0", GitTreeState:"clean", BuildDate:"2023-06-26T02:21:14Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/arm64"}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

StorageClasses:

NAME                                 PROVISIONER                       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
alicloud-disk-available              diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   557d
alicloud-disk-efficiency (default)   diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   557d
alicloud-disk-essd                   diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   557d
alicloud-disk-ssd                    diskplugin.csi.alibabacloud.com   Delete          Immediate              true                   557d
alicloud-disk-topology               diskplugin.csi.alibabacloud.com   Delete          WaitForFirstConsumer   true                   557d
alicloud-disk-topology-alltype       diskplugin.csi.alibabacloud.com   Delete          WaitForFirstConsumer   true                   557d

PVCs:

tikv-prod-redash-tikv-0                Bound     d-t4n4m5tgnop7v2qian23   1500Gi     RWO            alicloud-disk-topology-alltype   4d21h
tikv-prod-redash-tikv-1                Bound     d-t4nip5kxoy249mdxefow   1500Gi     RWO            alicloud-disk-topology-alltype   4d21h
tikv-prod-redash-tikv-10               Bound     d-t4n38r14xjl6kfribg3l   1800Gi     RWO            alicloud-disk-topology-alltype   521d
tikv-prod-redash-tikv-11               Bound     d-t4nh7zxh3g0tkdwomqm1   1800Gi     RWO            alicloud-disk-topology-alltype   518d
tikv-prod-redash-tikv-12               Bound     d-t4n1wa6dqfemdg3dmdme   1800Gi     RWO            alicloud-disk-topology-alltype   518d
tikv-prod-redash-tikv-13               Bound     d-t4n030w32k3vrtsstbla   1800Gi     RWO            alicloud-disk-topology-alltype   518d
tikv-prod-redash-tikv-14               Bound     d-t4ner7z7lw8mukulptxb   1800Gi     RWO            alicloud-disk-topology-alltype   340d
tikv-prod-redash-tikv-15               Bound     d-t4nf1osbhztaa8ct3id3   1800Gi     RWO            alicloud-disk-topology-alltype   104d
tikv-prod-redash-tikv-16               Pending                                                      alicloud-disk-topology-alltype   4d21h
tikv-prod-redash-tikv-17               Pending                                                      alicloud-disk-topology-alltype   4d21h
tikv-prod-redash-tikv-2                Bound     d-t4n4hmsh6g4cio235r83   1500Gi     RWO            alicloud-disk-topology-alltype   4d21h
tikv-prod-redash-tikv-3                Bound     d-t4n1e280p6kqmsvrhhyw   1800Gi     RWO            alicloud-disk-topology-alltype   529d
tikv-prod-redash-tikv-4                Bound     d-t4ni68o0s2kb9mykou93   1800Gi     RWO            alicloud-disk-topology-alltype   529d
tikv-prod-redash-tikv-5                Bound     d-t4n21ikxof96qumbgya1   1800Gi     RWO            alicloud-disk-topology-alltype   529d
tikv-prod-redash-tikv-6                Bound     d-t4n2vf6dxr6m8b4i4vbh   1800Gi     RWO            alicloud-disk-topology-alltype   529d
tikv-prod-redash-tikv-7                Bound     d-t4nbsfny2g2q4uhd3dhw   1800Gi     RWO            alicloud-disk-topology-alltype   522d
tikv-prod-redash-tikv-8                Bound     d-t4nfmfhdp5p4acq6mxp5   1800Gi     RWO            alicloud-disk-topology-alltype   521d
tikv-prod-redash-tikv-9                Bound     d-t4nil0s0f10guflgq6cw   1800Gi     RWO            alicloud-disk-topology-alltype   521d

What's the status of the TiDB cluster pods?

Status of kind TidbCluster, .status.tikv:

  tikv:
    bootStrapped: true
    image: pingcap/tikv:v7.5.1
    phase: Scale
    statefulSet:
      collisionCount: 0
      currentReplicas: 14
      currentRevision: prod-redash-tikv-58bf4d69df
      observedGeneration: 284
      readyReplicas: 14
      replicas: 14
      updateRevision: prod-redash-tikv-58bf4d69df
      updatedReplicas: 14
    stores:
      "1729778":
        id: "1729778"
        ip: prod-redash-tikv-3.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-10T15:37:47Z"
        leaderCount: 33239
        podName: prod-redash-tikv-3
        state: Up
      "2829612":
        id: "2829612"
        ip: prod-redash-tikv-4.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-07T19:33:09Z"
        leaderCount: 33239
        podName: prod-redash-tikv-4
        state: Up
      "4156292":
        id: "4156292"
        ip: prod-redash-tikv-5.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-10T15:39:23Z"
        leaderCount: 33233
        podName: prod-redash-tikv-5
        state: Up
      "4179344":
        id: "4179344"
        ip: prod-redash-tikv-6.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-10T15:37:17Z"
        leaderCount: 33236
        podName: prod-redash-tikv-6
        state: Up
      "1271929623":
        id: "1271929623"
        ip: prod-redash-tikv-7.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-07T19:45:07Z"
        leaderCount: 33237
        podName: prod-redash-tikv-7
        state: Up
      "1272506199":
        id: "1272506199"
        ip: prod-redash-tikv-8.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-07T19:45:07Z"
        leaderCount: 33238
        podName: prod-redash-tikv-8
        state: Up
      "1272575884":
        id: "1272575884"
        ip: prod-redash-tikv-9.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-10T15:41:26Z"
        leaderCount: 33237
        podName: prod-redash-tikv-9
        state: Up
      "1272575957":
        id: "1272575957"
        ip: prod-redash-tikv-10.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-10T15:41:21Z"
        leaderCount: 33240
        podName: prod-redash-tikv-10
        state: Up
      "1495134941":
        id: "1495134941"
        ip: prod-redash-tikv-11.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-07T19:43:50Z"
        leaderCount: 33238
        podName: prod-redash-tikv-11
        state: Up
      "1563719691":
        id: "1563719691"
        ip: prod-redash-tikv-12.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-10T15:39:20Z"
        leaderCount: 33236
        podName: prod-redash-tikv-12
        state: Up
      "1563727636":
        id: "1563727636"
        ip: prod-redash-tikv-13.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-12T14:40:17Z"
        leaderCount: 2491
        podName: prod-redash-tikv-13
        state: Offline
      "2986667096":
        id: "2986667096"
        ip: prod-redash-tikv-1.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-09T05:29:07Z"
        leaderCount: 33238
        podName: prod-redash-tikv-1
        state: Up
      "2986667099":
        id: "2986667099"
        ip: prod-redash-tikv-2.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-09T05:29:08Z"
        leaderCount: 33239
        podName: prod-redash-tikv-2
        state: Up
      "2986667162":
        id: "2986667162"
        ip: prod-redash-tikv-0.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: "2025-01-09T05:29:35Z"
        leaderCount: 33238
        podName: prod-redash-tikv-0
        state: Up
    synced: true
    tombstoneStores:
      "2279292722":
        id: "2279292722"
        ip: prod-redash-tikv-14.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: null
        leaderCount: 0
        podName: prod-redash-tikv-14
        state: Tombstone
      "2963433803":
        id: "2963433803"
        ip: prod-redash-tikv-15.prod-redash-tikv-peer.prod-tidb.svc
        lastTransitionTime: null
        leaderCount: 0
        podName: prod-redash-tikv-15
        state: Tombstone
    volumes:
      tikv:
        boundCount: 14
        currentCapacity: 1800Gi
        currentCount: 14
        currentStorageClass: alicloud-disk-topology-alltype
        modifiedCapacity: 1900Gi
        modifiedCount: 0
        modifiedStorageClass: alicloud-disk-topology-alltype
        name: tikv
        resizedCapacity: 1900Gi
        resizedCount: 0

What did you do?

  • Tidb-operator installed via Helm
  • Modify kind TidbCluster .spec.tikv.requests.storage: (1800Gi -> 1900Gi)

What did you expect to see?

  • I expect all Tikv PVCs to be resized to 1900Gi via the CSI provisioner in Alicloud ACK.

What did you see instead?

  • PVCs are not resized.
  • Could it be an issue with integrating Alicloud ACK CSI provisioner?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the change from 1800Gi to 1900Gi in TidbCluster .spec.tikv.requests.storage on Kubernetes v1.26.15 with the listed Alicloud storage class, then inspect the PVCs and .status.tikv.volumes fields. Use the reported kubectl commands and PVC events to compare the operator and CSI behavior. Done means the TiKV PVCs resize to 1900Gi and the status reflects the updated capacity.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.