Altinity / Altinity/clickhouse-operator

Operator does not wait for table part load before restarting other replicas

Open
#1,682 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

In many cases, the operator does rolling restart of all replicas. For example, I had the following replica configuration

                - templates:
                    podTemplate: cubes-pod-template
                - name: "4-b"
                  templates:
                    podTemplate: cubes-pod-template-b
                    dataVolumeClaimTemplate: cubes-dv-template-2t
                - name: "5-b"
                  templates:
                    podTemplate: cubes-pod-template-b
                    dataVolumeClaimTemplate: cubes-dv-template-2t

If I remove the first replica, the operator/K8S restarts the two remaining ones - 4-b and 5.b.

When using operator 0.24.5 and clickhouse 24.6.2.17 I observe that on pod restart, it reads table data. So 4-b replica has this in the log

│ clickhouse 2025.04.16 09:32:39.569252 [ 759 ] {} <Trace> cube.ads_stats_ad_sources_v5: Finished loading Active part 9927ba64533d0de103420ed52224dedf_1857_1857_0 on disk defau │
│ clickhouse 2025.04.16 09:32:39.569481 [ 759 ] {} <Trace> cube.ads_stats_ad_sources_v5: Loading Active part e1e4aed547c319c79e5ad19428b08454_803_803_0 from disk default

We have a lot of tables/parts, so the process takes several minutes. While doing that, it does not serve requests - if I get a shell into the pod and run any query, it just hangs. Queries over network likewise hang.

However, the operator proceeds to restart replica 5-b. At this point, both 4-b and 5-b are doing this initial table load, and neither responds to any requests.

Would it be possible to wait until each restarted replica is fully ready before restarting others?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the rolling restart with the three-replica configuration in the issue and review the operator's restart/readiness handling alongside the ClickHouse startup logs. Done means a restarted replica is confirmed ready to serve requests before the operator restarts the next replica, with coverage for this sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse, go, kubernetes
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.