Altinity / Altinity/clickhouse-operator
Operator does not wait for table part load before restarting other replicas
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
In many cases, the operator does rolling restart of all replicas. For example, I had the following replica configuration
- templates:
podTemplate: cubes-pod-template
- name: "4-b"
templates:
podTemplate: cubes-pod-template-b
dataVolumeClaimTemplate: cubes-dv-template-2t
- name: "5-b"
templates:
podTemplate: cubes-pod-template-b
dataVolumeClaimTemplate: cubes-dv-template-2t
If I remove the first replica, the operator/K8S restarts the two remaining ones - 4-b and 5.b.
When using operator 0.24.5 and clickhouse 24.6.2.17 I observe that on pod restart, it reads table data. So 4-b replica has this in the log
│ clickhouse 2025.04.16 09:32:39.569252 [ 759 ] {} <Trace> cube.ads_stats_ad_sources_v5: Finished loading Active part 9927ba64533d0de103420ed52224dedf_1857_1857_0 on disk defau │
│ clickhouse 2025.04.16 09:32:39.569481 [ 759 ] {} <Trace> cube.ads_stats_ad_sources_v5: Loading Active part e1e4aed547c319c79e5ad19428b08454_803_803_0 from disk default
We have a lot of tables/parts, so the process takes several minutes. While doing that, it does not serve requests - if I get a shell into the pod and run any query, it just hangs. Queries over network likewise hang.
However, the operator proceeds to restart replica 5-b. At this point, both 4-b and 5-b are doing this initial table load, and neither responds to any requests.
Would it be possible to wait until each restarted replica is fully ready before restarting others?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the rolling restart with the three-replica configuration in the issue and review the operator's restart/readiness handling alongside the ClickHouse startup logs. Done means a restarted replica is confirmed ready to serve requests before the operator restarts the next replica, with coverage for this sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse, go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100