cockroachdb / cockroachdb/cockroach-operator

Allow k8s operator users to add labels to existing deployments

Open
#993 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
318
Forks
104
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Adding a label to an existing operator deployment is not currently supported by our operator. It appears that adding a label, has been deemed restricted by the operator, reason this fails. Adding labels should not be considered a restricted op, however, according to the logs, this is forbidden:

```
spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
```

Steps to replicate:

The cluster below was deployed using the default operator and `example.yml` file. The `additionalLabels` line was present initially when the cluster was first deployed, so the label is applied correctly. Snippet below shows the line for the label and it is correctly applied:

```
..
...
....
..... snip ....
image:
name: cockroachdb/cockroach:v21.2.8
# nodes refers to the number of crdb pods that are created
# via the statefulset
nodes: 3
additionalLabels:
crdb: test-operator-deployment
```

pod details, label is applied as expected and pods are ready

```
kubectl get pods -n cockroach-operator-system -o wide --show-labels
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES LABELS
cluster-init-ndl5z 1/1 Running 0 2m44s 10.244.2.11 lab-kub03 controller-uid=0636d470-26da-40b0-a2db-ac4a2c67d847,job-name=cluster-init
cockroach-operator-manager-df946bb6b-gfdzl 1/1 Running 0 3m 10.244.1.223 lab-kub02 app=cockroach-operator,pod-template-hash=df946bb6b
cockroachdb-0 1/1 Running 0 2m45s 10.244.2.13 lab-kub03 app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-84ccf97766,crdb=test-operator-deployment,statefulset.kubernetes.io/pod-name=cockroachdb-0
cockroachdb-1 1/1 Running 0 2m45s 10.244.1.225 lab-kub02 app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-84ccf97766,crdb=test-operator-deployment,statefulset.kubernetes.io/pod-name=cockroachdb-1
cockroachdb-2 1/1 Running 0 2m45s 10.244.2.14 lab-kub03 app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-84ccf97766,crdb=test-operator-deployment,statefulset.kubernetes.io/pod-name=cockroachdb-2
```

I then destroyed the cluster and commented out the section where the label is defined, so the pods would only have the standard default labels:

```
kubectl get pods -n $ns --show-labels
NAME READY STATUS RESTARTS AGE LABELS
cluster-init-v56ph 1/1 Running 0 64s controller-uid=1ab53c79-18c2-4506-a0d7-10aa9fcaa918,job-name=cluster-init
cockroach-operator-manager-df946bb6b-vvwhd 1/1 Running 0 80s app=cockroach-operator,pod-template-hash=df946bb6b
cockroachdb-0 1/1 Running 0 66s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-6478ffd568,statefulset.kubernetes.io/pod-name=cockroachdb-0
cockroachdb-1 1/1 Running 0 66s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-6478ffd568,statefulset.kubernetes.io/pod-name=cockroachdb-1
cockroachdb-2 1/1 Running 0 66s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-6478ffd568,statefulset.kubernetes.io/pod-name=cockroachdb-2
enterprise-license-qcwc2 0/1 Completed 0 64s controller-uid=9c705366-b3e9-4c46-a595-b38fad521565,job-name=enterprise-license
```

As you can see above, all pods are running and the default labels were applied as expected.
I then modified the `CrdbCluster` definition in the file `example.yml` to include a new label, snippet of the pertaining section below:

```
image:
name: cockroachdb/cockroach:v22.1.0
# nodes refers to the number of crdb pods that are created
# via the statefulset
maxUnavailable: 1
minAvailable: 2
nodes: 3
additionalLabels:
daniel: custom-test-lab-operator-deployment
```

I then applied the `example.yml` file:

```
kubectl apply -f example.yml
crdbcluster.crdb.cockroachlabs.com/cockroachdb configured
```

The pods still running at this point:

```
kubectl get pods -n $ns -w
NAME READY STATUS RESTARTS AGE
cluster-init-v56ph 1/1 Running 0 4m12s
cockroach-operator-manager-df946bb6b-vvwhd 1/1 Running 0 4m28s
cockroachdb-0 1/1 Running 0 4m14s
cockroachdb-1 1/1 Running 0 4m14s
cockroachdb-2 1/1 Running 0 4m14s
```

Now I do a rolling restart:

```
kubectl rollout restart statefulset cockroachdb -n $ns
statefulset.apps/cockroachdb restarted
```

and monitor the status, which will show the 3rd pod restarting and getting stuck as shown below and will not proceed:

```
kubectl get pods -n $ns -w
NAME READY STATUS RESTARTS AGE
cluster-init-v56ph 1/1 Running 0 4m12s
cockroach-operator-manager-df946bb6b-vvwhd 1/1 Running 0 4m28s
cockroachdb-0 1/1 Running 0 4m14s
cockroachdb-1 1/1 Running 0 4m14s
cockroachdb-2 1/1 Running 0 4m14s
cockroachdb-2 1/1 Terminating 0 5m55s
cockroachdb-2 0/1 Terminating 0 6m11s
cockroachdb-2 0/1 Terminating 0 6m11s
cockroachdb-2 0/1 Terminating 0 6m11s
cockroachdb-2 0/1 Pending 0 0s
cockroachdb-2 0/1 Pending 0 0s
cockroachdb-2 0/1 ContainerCreating 0 1s
cockroachdb-2 0/1 Running 0 15s
```

Notice the new label was not applied yet and the pod status showing it is stuck:

```
kubectl get pods -n $ns --show-labels
NAME READY STATUS RESTARTS AGE LABELS
cluster-init-v56ph 1/1 Running 0 9m41s controller-uid=1ab53c79-18c2-4506-a0d7-10aa9fcaa918,job-name=cluster-init
cockroach-operator-manager-df946bb6b-vvwhd 1/1 Running 0 9m57s app=cockroach-operator,pod-template-hash=df946bb6b
cockroachdb-0 1/1 Running 0 9m43s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-6478ffd568,statefulset.kubernetes.io/pod-name=cockroachdb-0
cockroachdb-1 1/1 Running 0 9m43s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-6478ffd568,statefulset.kubernetes.io/pod-name=cockroachdb-1
cockroachdb-2 0/1 Running 0 3m32s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-b87596cd9,statefulset.kubernetes.io/pod-name=cockroachdb-2
```

I then went ahead and commented out the line from the `example.yml` that configures the label, snippet shown below:

```
image:
name: cockroachdb/cockroach:v22.1.0
# nodes refers to the number of crdb pods that are created
# via the statefulset
maxUnavailable: 1
minAvailable: 2
nodes: 3
# additionalLabels:
# daniel: custom-test-lab-operator-deployment
```

I then applied the changed file:

```
kubectl apply -f example.yml
crdbcluster.crdb.cockroachlabs.com/cockroachdb configured
```

I waited a few seconds after the new file was applied above and the pods proceeded to restart successfully. Notice full output below:

```
kubectl get pods -n $ns -w
NAME READY STATUS RESTARTS AGE
cluster-init-v56ph 1/1 Running 0 4m12s
cockroach-operator-manager-df946bb6b-vvwhd 1/1 Running 0 4m28s
cockroachdb-0 1/1 Running 0 4m14s
cockroachdb-1 1/1 Running 0 4m14s
cockroachdb-2 1/1 Running 0 4m14s
cockroachdb-2 1/1 Terminating 0 5m55s
cockroachdb-2 0/1 Terminating 0 6m11s
cockroachdb-2 0/1 Terminating 0 6m11s
cockroachdb-2 0/1 Terminating 0 6m11s
cockroachdb-2 0/1 Pending 0 0s
cockroachdb-2 0/1 Pending 0 0s
cockroachdb-2 0/1 ContainerCreating 0 1s
cockroachdb-2 0/1 Running 0 15s
cockroachdb-2 1/1 Running 0 6m26s
cockroachdb-1 1/1 Terminating 0 12m
cockroachdb-1 0/1 Terminating 0 12m
cockroachdb-1 0/1 Terminating 0 12m
cockroachdb-1 0/1 Terminating 0 12m
cockroachdb-1 0/1 Pending 0 0s
cockroachdb-1 0/1 Pending 0 0s
cockroachdb-1 0/1 ContainerCreating 0 0s
cockroachdb-1 0/1 Running 0 3s
cockroachdb-1 1/1 Running 0 45s
cockroachdb-0 1/1 Terminating 0 13m
cockroachdb-0 0/1 Terminating 0 13m
cockroachdb-0 0/1 Terminating 0 13m
cockroachdb-0 0/1 Terminating 0 13m
cockroachdb-0 0/1 Pending 0 0s
cockroachdb-0 0/1 Pending 0 0s
cockroachdb-0 0/1 ContainerCreating 0 0s
cockroachdb-0 0/1 Running 0 3s
cockroachdb-0 1/1 Running 0 30s
```

and now the final output, no custom label applied since the line with the new label has been commented out, but the all pods are running:

```
kubectl get pods -n $ns --show-labels
NAME READY STATUS RESTARTS AGE LABELS
cluster-init-v56ph 1/1 Running 0 14m controller-uid=1ab53c79-18c2-4506-a0d7-10aa9fcaa918,job-name=cluster-init
cockroach-operator-manager-df946bb6b-vvwhd 1/1 Running 0 15m app=cockroach-operator,pod-template-hash=df946bb6b
cockroachdb-0 1/1 Running 0 68s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-b87596cd9,statefulset.kubernetes.io/pod-name=cockroachdb-0
cockroachdb-1 1/1 Running 0 2m6s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-b87596cd9,statefulset.kubernetes.io/pod-name=cockroachdb-1
cockroachdb-2 1/1 Running 0 8m49s app.kubernetes.io/component=database,app.kubernetes.io/instance=cockroachdb,app.kubernetes.io/name=cockroachdb,controller-revision-hash=cockroachdb-b87596cd9,statefulset.kubernetes.io/pod-name=cockroachdb-2
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.