Altinity / Altinity/clickhouse-operator

CHI StatfulSets deleted after reconciliation

Open
#1,907 7 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

fixed
Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

I recently experienced a situation where, after upgrading clickhouse-operator from 0.25.3 to 0.25.6, the operator ended up deleting both StatefulSet resources from my cluster (1 shard 2 replicas) while reconciling changes. Another similar situation occurred where the operator deleted one of the StatefulSet resources and managed to reconcile/update the other.

You can find the logs from clickhouse-operator in this gist

I started looking into what could have caused the issue with Claude's help, but i don't fully trust the conclusions. Claude also wrote up a "bug report", which I have cleaned up a little and pasted into this gist. If you prefer, I can update this ticket and inline the contents of claude's "bug report" here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Review the clickhouse-operator logs and the cleaned-up bug report in the linked gists, then trace the upgrade reconciliation involving the two StatefulSet resources. Confirm the deletion scenario and establish regression coverage showing that both StatefulSets remain managed after reconciliation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.