Altinity / Altinity/clickhouse-operator
buildCR issues one uncached StatefulSet GET per host (twice per reconcile) instead of a single List/cache
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Problem
On every reconcile, buildCR walks all hosts and reads each host's current StatefulSet individually:
fillCurSTS→STS().Get(ctx, host)for every hostfindMinMaxVersions→PrepareHostStatefulSetWithStatus→getStatefulSetStatus→STS().Get(...)for every host
Each read is a live call to the API server (pkg/controller/chi/kube/statesfulset.go): kubeClient.AppsV1().StatefulSets(ns).Get(..., NewGetOptions()), where NewGetOptions() returns an empty metav1.GetOptions{} — i.e. no ResourceVersion, so it's a full read from etcd rather than a cached read. The host walk also runs twice per buildCR (once before pod-IP collection, once after).
So a single reconcile does on the order of 4 × hostCount sequential StatefulSet GETs before it reconciles the first host.
Impact
On a several-hundred-host CHI this "build" pre-pass is latency-bound and can take minutes on its own, before any host reconcile begins. It also adds etcd/apiserver read load that grows linearly with host count. Together with #2045, it contributes to large installations appearing to make little progress, since a good share of the wall-clock is spent re-reading state that could be fetched in one call or served from cache.
Fix ideas
- Read StatefulSets from the informer/lister cache the operator already maintains, instead of the live clientset.
- Or replace the per-host GETs with a single labeled
Listper namespace into a map, and look up per host (a miss then means genuinely absent, which still detects deleted StatefulSets correctly). - Alternatively, use a cached read (
GetOptions{ResourceVersion: "0"}, served from the apiserver watch cache), which avoids the etcd round-trip per call.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/controller/chi/kube/statesfulset.go, then trace buildCR through fillCurSTS and findMinMaxVersions to confirm where StatefulSet GETs are issued. Compare the informer/lister, labeled List, and cached-read options described in the issue, and preserve detection of genuinely absent StatefulSets. Done means the reconcile no longer performs repeated uncached per-host reads while retaining correct host status handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100