Altinity / Altinity/clickhouse-operator
Per-host status writes rewrite the full CHI .status, causing repeated update conflicts under reconcile concurrency
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Problem
As each host finishes reconciling, the operator writes the entire CHI .status back with a read-modify-write (pkg/controller/chi/kube/cr.go, statusUpdateProcess): Get the current CHI → EnsureStatus().CopyFrom(...) → full Update, wrapped in statusUpdateRetry (up to maxRetryAttempts = 50, with a sleep between attempts).
When many hosts reconcile concurrently, they all update the same single CHI object. Kubernetes allows only one update to an object at a time (optimistic concurrency), so the others get:
Operation cannot be fulfilled on clickhouseinstallations.clickhouse.altinity.com "<chi>":
the object has been modified; please apply your changes to the latest version and try again
…and then re-Get and retry. Under concurrency this leads to repeated write conflicts: a single host's status write can take several attempts (observed attempt 2, 3, 4, 6, …) with backoff between each.
Impact
At higher reconcile concurrency (>50) the per-host status write can become a significant per-host cost (observed in the tens of seconds), most of it spent on conflict retries and backoff rather than the update itself. It also results in repeated full-object writes to the apiserver. Because the contention comes from concurrency, the cost tends to increase as reconcile concurrency is raised.
Fix ideas
- Use a status subresource
Patch(JSON/strategic-merge patch of just the changed fields) instead of a read-modify-write of the whole.status. - Or coalesce/batch host-completion status updates rather than one full-object write per completing host (for example, a single writer that flushes accumulated per-host transitions).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/controller/chi/kube/cr.go at statusUpdateProcess and statusUpdateRetry, then trace how concurrent host reconciles update the CHI status subresource. Reproduce the conflict behavior with high reconcile concurrency and compare the proposed patch or coalescing paths. Done means host status updates avoid repeated full-object conflicts and preserve all required status transitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100