Altinity / Altinity/clickhouse-operator

ClickHouseKeeperInstallation stuck at status: InProgress despite healthy 3-node quorum (operator 0.27.0)

Open
#2,035 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Keeper
Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Description

With altinity-clickhouse-operator 0.27.0, a ClickHouseKeeperInstallation (CHK) never transitions to status: Completed. It stays InProgress indefinitely even though the Keeper cluster is fully healthy, and the operator reconcile-loops on the CHK every ~10s without converging. A sibling ClickHouseInstallation (CHI) managed by the same operator reconciles to Completed normally.

Environment
  • clickhouse-operator: 0.27.0 (chart altinity-clickhouse-operator-0.27.0)
  • Kubernetes: 1.35 (GKE)
  • ClickHouse Keeper image: 26.5.x, 1 shard / 3 replicas
What's healthy

The Keeper has a full quorum — from mntr on a node:

zk_server_state        leader
zk_followers           2
zk_synced_followers    2

All 3 keeper pods are Running/Ready, the StatefulSets report rolling-update complete, and the ClickHouseInstallation (server) that uses this Keeper reaches status: Completed.

Observed behavior

The CHK stays status: InProgress; taskIDsStarted keeps growing while taskIDsCompleted stays at 0. The operator repeats this cycle continuously:

worker-reconciler-chk.go:57  reconcileCR():start
object-status.go:47          GetObjectStatusFromMetas(): cur and new objects are equal ... Update of the object is not required
worker-reconciler-chk.go:795 prepareStsReconcileOptsWaitSection(): Setting option SetWaitUntilReady   (once per host)
worker-reconciler-chk.go:91  reconcileCR(): CR has reconcile work - continue reconcile
worker-reconciler-chk.go:132 reconcileCR():end

It reproduces on a freshly created cluster, so it is not tied to any pre-existing or corrupted state.

Expected

Once the Keeper StatefulSets are ready and the quorum is healthy, the CHK should reach status: Completed (as the CHI does).

Questions
  • Is there a readiness/coordination gate (e.g. reconcile.coordination.keeper.readyTimeout) that the CHK controller waits on and never satisfies in this scenario?
  • Any known issue with CHK status finalization in 0.27.0?

Happy to add a minimal reproducer manifest and full operator logs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with worker-reconciler-chk.go, especially reconcileCR() and prepareStsReconcileOptsWaitSection(), then inspect object-status.go around GetObjectStatusFromMetas(). Compare the CHK reconciliation and status finalization path with the sibling CHI behavior described in the report. Done means a healthy three-replica Keeper reaches status Completed without repeated reconciliation or growing taskIDsStarted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.