pingcap / pingcap/tidb-operator

Empty objects under `storageClaims` bypass TiDB’s validation

Open
#4,613 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.3k
Forks
540
Avg merge
3d 2h
Merged PRs (30d)
18

Description

Bug Report

What version of Kubernetes are you using?

Client Version: [version.Info](http://version.info/){Major:“1”, Minor:“17", GitVersion:“v1.17.1”, GitCommit:“d224476cd0730baca2b6e357d144171ed74192d6", GitTreeState:“clean”, BuildDate:“2020-01-14T21:04:32Z”, GoVersion:“go1.13.5”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: [version.Info](http://version.info/){Major:“1", Minor:“22”, GitVersion:“v1.22.9", GitCommit:“6df4433e288edc9c40c2e344eb336f63fad45cd2”, GitTreeState:“clean”, BuildDate:“2022-05-19T19:53:08Z”, GoVersion:“go1.16.15", Compiler:“gc”, Platform:“linux/amd64"}

What version of TiDB Operator are you using?

TiDB Operator Version: [version.Info](http://version.info/){GitVersion:“v1.3.0-45+1470cfb46e1ffb-dirty”, GitCommit:“1470cfb46e1ffb8bb86f74ba455865a95b825413", GitTreeState:“dirty”, BuildDate:“2022-07-06T18:55:00Z”, GoVersion:“go1.18.3”, Compiler:“gc”, Platform:“linux/amd64”}

What storage classes exist in the Kubernetes cluster and what are used for PD/TiKV pods?

$ kubectl get sc
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   [rancher.io/local-path](http://rancher.io/local-path)   Delete          WaitForFirstConsumer   false                  40m

$ kubectl get pvc -n {tidb-cluster-namespace}
NAME                        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pd-advanced-tidb-pd-0       Bound    pvc-b566858b-bf4e-4e33-b31e-5d7feb7397b1   10Gi       RWO            standard       40m
pd-advanced-tidb-pd-1       Bound    pvc-df70980f-12cf-499f-8ad7-e41cac98c5d0   10Gi       RWO            standard       40m
pd-advanced-tidb-pd-2       Bound    pvc-d41691d8-feb5-4e21-b282-1ece1851cffa   10Gi       RWO            standard       40m
tikv-advanced-tidb-tikv-0   Bound    pvc-42e652d6-2400-4ae8-b790-cef8466e4566   100Gi      RWO            standard       39m
tikv-advanced-tidb-tikv-1   Bound    pvc-5af08c43-e02d-433c-896a-b85ad568d1ca   100Gi      RWO            standard       39m
tikv-advanced-tidb-tikv-2   Bound    pvc-652761b6-9fff-4080-b13d-9e364062cddc   100Gi      RWO            standard       39m

What’s the status of the TiDB cluster pods?

$ kubectl get po -n {tidb-cluster-namespace} -o wide
NAME                                       READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
advanced-tidb-discovery-6998694d4c-fmpjl   1/1     Running   0          41m   10.244.2.2   test-worker3   <none>           <none>
advanced-tidb-pd-0                         1/1     Running   0          41m   10.244.3.4   test-worker    <none>           <none>
advanced-tidb-pd-1                         1/1     Running   0          41m   10.244.2.4   test-worker3   <none>           <none>
advanced-tidb-pd-2                         1/1     Running   0          41m   10.244.1.4   test-worker2   <none>           <none>
advanced-tidb-tidb-0                       2/2     Running   0          39m   10.244.3.7   test-worker    <none>           <none>
advanced-tidb-tidb-1                       2/2     Running   0          39m   10.244.2.7   test-worker3   <none>           <none>
advanced-tidb-tidb-2                       2/2     Running   0          39m   10.244.1.7   test-worker2   <none>           <none>
advanced-tidb-tikv-0                       1/1     Running   0          40m   10.244.2.6   test-worker3   <none>           <none>
advanced-tidb-tikv-1                       1/1     Running   0          40m   10.244.3.6   test-worker    <none>           <none>
advanced-tidb-tikv-2                       1/1     Running   0          40m   10.244.1.6   test-worker2   <none>           <none>
tidb-controller-manager-6cc68f8949-52vwt   1/1     Running   0          41m   10.244.3.2   test-worker    <none>           <none>
tidb-scheduler-dd569b6b4-hj294             2/2     Running   0          41m   10.244.1.2   test-worker2   <none>           <none>

What did you do?

We created a CR with TiFlash as follows and we deployed it using kubectl apply -f.

The CR file
apiVersion: [pingcap.com/v1alpha1](http://pingcap.com/v1alpha1)
kind: TidbCluster
metadata:
  name: advanced-tidb
spec:
  version: “v5.4.0"
  timezone: UTC
  configUpdateStrategy: RollingUpdate
  helper:
    image: busybox:1.34.1
  pvReclaimPolicy: Retain
  enableDynamicConfiguration: true
  tiflash:
    baseImage: pingcap/tiflash
    replicas: 2
    storageClaims:
    - {}
    - {}
    - {}
  pd:
    baseImage: pingcap/pd
    config: |
      [dashboard]
        internal-proxy = true
    replicas: 3
    maxFailoverCount: 0
    requests:
      storage: 10Gi
    mountClusterClientSecret: true
  tidb:
    baseImage: pingcap/tidb
    config: |
      [performance]
        tcp-keep-alive = true
    replicas: 3
    maxFailoverCount: 0
    service:
      type: NodePort
      externalTrafficPolicy: Local
  tikv:
    baseImage: pingcap/tikv
    config: |
      log-level = “info”
    replicas: 3
    maxFailoverCount: 0
    requests:
      storage: 100Gi
    mountClusterClientSecret: true

Note that we assigned empty values to storageClaims.

What did you expect to see?
We expected to see that the input would be rejected as storageClaims is essentially empty. Additionally, we believed that we should see an error message “storageClaims should be configured at least one item.” as this line suggests.

What did you see instead?
The pod related to TiFlash was never created but we did not see the error message that we mentioned above. In fact, we did not see any error messages from the operator indicating why the pod for TiFlash failed to be created.

Possible reason
We believed that the validation here is not comprehensive since even if we build storageClaims with empty values, it will bypass this if statement as the length of spec.StorageClaims is 3 in our case.

Besides, from kubectl we found out that spec.storageClaims[INDEX].resources.requests.storage is a required field. But it is labelled as optional in types.go at this line, which should be incorrect.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/apis/pingcap/v1alpha1/validation/validation.go around the storageClaims length check, then compare the storage claim fields in pkg/apis/pingcap/v1alpha1/types.go. Reproduce the TidbCluster manifest with empty storageClaims entries using kubectl apply and inspect the resulting validation or operator behavior. Done means invalid empty entries are rejected with a useful error explaining the required storage configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.