cockroachdb / cockroachdb/cockroach-operator

Cluster restarts in sequence when config changes, rather than in parallel

Open
#1,082 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
318
Forks
104
Avg merge
1d 6h
Merged PRs (30d)
1

Description

I have a Helm chart containing a `CrdbCluster` resource.

This Helm chart creates certificate secrets (CA, node, root) then uses the names of those secrets in the `nodeTLSSecret` and `clientTLSSecrets` values of the `CrdbCluster` resource config.

If the secrets already exist (found using Helm's `lookup` function), the existing certificate data is used. In this way, repeating the `helm install` or `helm upgrade` commands won't keep creating new certificates.

However, if I manually delete the CA certificate secret, the chart will generate all-new certificate secrets, albeit with the same names.
This means that the `CrdbCluster` config does not change, so the DB pods remain running with the old certificate data in their `cockroach-certs` folders.
This is not what I'm aiming for. I want the DB pods to be rebuilt when the certificates change.

So I added some `additionalAnnotations` to the `CrdbCluster` resource, containing the SHA1 hashes of the certificate data.
This looked like it was going to work, but something (the operator?) is attempting to restart the cluster pods **one by one, rather than all at once**.
So the first cluster pod restarts (with the new certificate data contained within), and doesn't report as "healthy" until it has successfully contacted the other cluster pods, **which it fails to do, as they still contain the old certificate data**. The operator doesn't seem to want to move on rebuilding the next DB pod until this first one reports as healthy.
The only way I can get them **all** to restart is to manually delete them.

Is it possible for the operator to restart them all at once?
(actually, I'm assuming it's the operator that's controlling this, and not some fundamental k8s component ... maybe you could confirm this)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.