Altinity / Altinity/clickhouse-operator

buildCR issues one uncached StatefulSet GET per host (twice per reconcile) instead of a single List/cache

Open
#2,063 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

big deployment planned for review
Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Problem

On every reconcile, buildCR walks all hosts and reads each host's current StatefulSet individually:

  • fillCurSTSSTS().Get(ctx, host) for every host
  • findMinMaxVersionsPrepareHostStatefulSetWithStatusgetStatefulSetStatusSTS().Get(...) for every host

Each read is a live call to the API server (pkg/controller/chi/kube/statesfulset.go): kubeClient.AppsV1().StatefulSets(ns).Get(..., NewGetOptions()), where NewGetOptions() returns an empty metav1.GetOptions{} — i.e. no ResourceVersion, so it's a full read from etcd rather than a cached read. The host walk also runs twice per buildCR (once before pod-IP collection, once after).

So a single reconcile does on the order of 4 × hostCount sequential StatefulSet GETs before it reconciles the first host.

Impact

On a several-hundred-host CHI this "build" pre-pass is latency-bound and can take minutes on its own, before any host reconcile begins. It also adds etcd/apiserver read load that grows linearly with host count. Together with #2045, it contributes to large installations appearing to make little progress, since a good share of the wall-clock is spent re-reading state that could be fetched in one call or served from cache.

Fix ideas
  • Read StatefulSets from the informer/lister cache the operator already maintains, instead of the live clientset.
  • Or replace the per-host GETs with a single labeled List per namespace into a map, and look up per host (a miss then means genuinely absent, which still detects deleted StatefulSets correctly).
  • Alternatively, use a cached read (GetOptions{ResourceVersion: "0"}, served from the apiserver watch cache), which avoids the etcd round-trip per call.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/controller/chi/kube/statesfulset.go, then trace buildCR through fillCurSTS and findMinMaxVersions to confirm where StatefulSet GETs are issued. Compare the informer/lister, labeled List, and cached-read options described in the issue, and preserve detection of genuinely absent StatefulSets. Done means the reconcile no longer performs repeated uncached per-host reads while retaining correct host status handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.