Altinity / Altinity/clickhouse-operator
Adding an invalid label causes whole cluster to be removed
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Operator version: 0.23.5
Adding an invalid label to a podtemplate, eventually causes the operator to delete all statefulsets during reconciliation, regardless of settings.
I have the following settings:
runtime:
reconcileCHIsThreadsNumber: 10
reconcileShardsThreadsNumber: 5
reconcileShardsMaxConcurrencyPercent: 50
threadsNumber: 0
statefulSet:
create:
onFailure: abort
update:
timeout: 300
pollInterval: 5
onFailure: rollback
host:
wait:
exclude: "true"
queries: "true"
include: "false"
After adding an invalid label to spec.templates.podTemplates[0].metadata.label e.g. some_bad_label: '/metrics' the operator tries to recreate the statefulsets but encounters the following error:
E0508 13:36:35.377188 1 creator.go:46] createStatefulSet():StatefulSet create failed. err: StatefulSet.apps "chi-clickhouse-store-0-0" is invalid: spec.template.labels: Invalid value: "/metrics": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue', or 'my_value', or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')
My expected behavior is:
After failing to create the statefulset the operator either aborts or rolls back
Actual behavior:
After some time period, the operator moves to the next statefulset until all are deleted (and not recreated due to error)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing reconciliation after the StatefulSet validation error, focusing on how create failures interact with the configured abort and rollback settings. Reproduce the invalid pod-template label case and verify that an unsuccessful StatefulSet creation does not cause existing StatefulSets to be deleted, while abort or rollback behavior is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100