Altinity / Altinity/clickhouse-operator

Per-host status writes rewrite the full CHI .status, causing repeated update conflicts under reconcile concurrency

Open
#2,064 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Problem

As each host finishes reconciling, the operator writes the entire CHI .status back with a read-modify-write (pkg/controller/chi/kube/cr.go, statusUpdateProcess): Get the current CHI → EnsureStatus().CopyFrom(...) → full Update, wrapped in statusUpdateRetry (up to maxRetryAttempts = 50, with a sleep between attempts).

When many hosts reconcile concurrently, they all update the same single CHI object. Kubernetes allows only one update to an object at a time (optimistic concurrency), so the others get:

Operation cannot be fulfilled on clickhouseinstallations.clickhouse.altinity.com "<chi>":
the object has been modified; please apply your changes to the latest version and try again

…and then re-Get and retry. Under concurrency this leads to repeated write conflicts: a single host's status write can take several attempts (observed attempt 2, 3, 4, 6, …) with backoff between each.

Impact

At higher reconcile concurrency (>50) the per-host status write can become a significant per-host cost (observed in the tens of seconds), most of it spent on conflict retries and backoff rather than the update itself. It also results in repeated full-object writes to the apiserver. Because the contention comes from concurrency, the cost tends to increase as reconcile concurrency is raised.

Fix ideas
  • Use a status subresource Patch (JSON/strategic-merge patch of just the changed fields) instead of a read-modify-write of the whole .status.
  • Or coalesce/batch host-completion status updates rather than one full-object write per completing host (for example, a single writer that flushes accumulated per-host transitions).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/controller/chi/kube/cr.go at statusUpdateProcess and statusUpdateRetry, then trace how concurrent host reconciles update the CHI status subresource. Reproduce the conflict behavior with high reconcile concurrency and compare the proposed patch or coalescing paths. Done means host status updates avoid repeated full-object conflicts and preserve all required status transitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.