Altinity / Altinity/clickhouse-operator
ClickHouseKeeperInstallation stuck at status: InProgress despite healthy 3-node quorum (operator 0.27.0)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Description
With altinity-clickhouse-operator 0.27.0, a ClickHouseKeeperInstallation (CHK) never transitions to status: Completed. It stays InProgress indefinitely even though the Keeper cluster is fully healthy, and the operator reconcile-loops on the CHK every ~10s without converging. A sibling ClickHouseInstallation (CHI) managed by the same operator reconciles to Completed normally.
Environment
- clickhouse-operator: 0.27.0 (chart
altinity-clickhouse-operator-0.27.0) - Kubernetes: 1.35 (GKE)
- ClickHouse Keeper image: 26.5.x, 1 shard / 3 replicas
What's healthy
The Keeper has a full quorum — from mntr on a node:
zk_server_state leader
zk_followers 2
zk_synced_followers 2
All 3 keeper pods are Running/Ready, the StatefulSets report rolling-update complete, and the ClickHouseInstallation (server) that uses this Keeper reaches status: Completed.
Observed behavior
The CHK stays status: InProgress; taskIDsStarted keeps growing while taskIDsCompleted stays at 0. The operator repeats this cycle continuously:
worker-reconciler-chk.go:57 reconcileCR():start
object-status.go:47 GetObjectStatusFromMetas(): cur and new objects are equal ... Update of the object is not required
worker-reconciler-chk.go:795 prepareStsReconcileOptsWaitSection(): Setting option SetWaitUntilReady (once per host)
worker-reconciler-chk.go:91 reconcileCR(): CR has reconcile work - continue reconcile
worker-reconciler-chk.go:132 reconcileCR():end
It reproduces on a freshly created cluster, so it is not tied to any pre-existing or corrupted state.
Expected
Once the Keeper StatefulSets are ready and the quorum is healthy, the CHK should reach status: Completed (as the CHI does).
Questions
- Is there a readiness/coordination gate (e.g.
reconcile.coordination.keeper.readyTimeout) that the CHK controller waits on and never satisfies in this scenario? - Any known issue with CHK status finalization in 0.27.0?
Happy to add a minimal reproducer manifest and full operator logs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with worker-reconciler-chk.go, especially reconcileCR() and prepareStsReconcileOptsWaitSection(), then inspect object-status.go around GetObjectStatusFromMetas(). Compare the CHK reconciliation and status finalization path with the sibling CHI behavior described in the report. Done means a healthy three-replica Keeper reaches status Completed without repeated reconciliation or growing taskIDsStarted.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100