Altinity / Altinity/clickhouse-operator

Adding an invalid label causes whole cluster to be removed

Open
#1,420 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question stale
Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Operator version: 0.23.5

Adding an invalid label to a podtemplate, eventually causes the operator to delete all statefulsets during reconciliation, regardless of settings.

I have the following settings:

    runtime:
        reconcileCHIsThreadsNumber: 10
        reconcileShardsThreadsNumber: 5
        reconcileShardsMaxConcurrencyPercent: 50
        threadsNumber: 0
    statefulSet:
        create:
            onFailure: abort
        update:
            timeout: 300
            pollInterval: 5
            onFailure: rollback
    host:
        wait:
            exclude: "true"
            queries: "true"
            include: "false"

After adding an invalid label to spec.templates.podTemplates[0].metadata.label e.g. some_bad_label: '/metrics' the operator tries to recreate the statefulsets but encounters the following error:

E0508 13:36:35.377188       1 creator.go:46] createStatefulSet():StatefulSet create failed. err: StatefulSet.apps "chi-clickhouse-store-0-0" is invalid: spec.template.labels: Invalid value: "/metrics": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

My expected behavior is:
After failing to create the statefulset the operator either aborts or rolls back

Actual behavior:
After some time period, the operator moves to the next statefulset until all are deleted (and not recreated due to error)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing reconciliation after the StatefulSet validation error, focusing on how create failures interact with the configured abort and rollback settings. Reproduce the invalid pod-template label case and verify that an unsuccessful StatefulSet creation does not cause existing StatefulSets to be deleted, while abort or rollback behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.