Altinity / Altinity/clickhouse-operator

Correct way to upgrade operator from 1.23.7 to 1.24+ with multiple clickhouse keeper replicas

Open
#1,740 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Hello
I am following https://github.com/Altinity/clickhouse-operator/blob/0.24.0/docs/keeper_migration_from_23_to_24.md instruction to upgrade operator from 1.23.7 to 1.24.5

Unfortunately I had encountered an issue during the upgrade of installation with 3 replicas of clickhouse keepers:
While with old operator version all keepers located in the single statefulset which has a single service; with 1.24.5 a separate statefulset and service is created for each replica. Which basically changes the addresses of all keeper replicas.
For example in my installation for version 1.23.7 I had 3 keeper addresses:
clickhouse-keeper-logging-0.clickhouse-keeper-logging-headless
clickhouse-keeper-logging-1.clickhouse-keeper-logging-headless
clickhouse-keeper-logging-2.clickhouse-keeper-logging-headless

And after the update they changed to:
chk-clickhouse-keeper-logging-default-0-0
chk-clickhouse-keeper-logging-default-1-0
chk-clickhouse-keeper-logging-default-2-0

I saw that <raft_configuration> was indeed changed to a correct one after an update:

<raft_configuration>
    <server>
        <id>0</id>
        <hostname>chk-clickhouse-keeper-logging-default-0-0</hostname>
        <port>9444</port>
    </server>
    <server>
        <id>1</id>
        <hostname>chk-clickhouse-keeper-logging-default-0-1</hostname>
        <port>9444</port>
    </server>
    <server>
        <id>2</id>
        <hostname>chk-clickhouse-keeper-logging-default-0-2</hostname>
        <port>9444</port>
    </server>
</raft_configuration>

However even having the updated raft_configuration, keeper replicas were unable to reach each other, as hosts stored in replication log were not updated in the process and replicas continued using old addresses

clickhouse-keeper-logging-1:/$ clickhouse-keeper client -h localhost --port 2181 -q "get '/keeper/config'"
server.0=clickhouse-keeper-logging-0.clickhouse-keeper-logging-headless:9444;participant;1
server.2=clickhouse-keeper-logging-2.clickhouse-keeper-logging-headless:9444;participant;1
server.1=clickhouse-keeper-logging-1.clickhouse-keeper-logging-headless:9444;participant;1

I found no easy way to make clickhouse keeper reload host configuration from disk or similar issues, but maybe I am missing something here?

What I tried:

  1. Configuring the cluster for keeper hosts to have the same addresses as before
    I failed to come to any solution without keeping and maintaining additional k8s objects which is not desirable, as configuration had changed and we now have 3 services for 3 keeper replicas instead of one.

  2. Adding new hosts using zookeeper reconfigure command
    Doesnt seem possible to incrementally change /keeper/config to the state matching raft_configuration, as we can not reuse server ids

  3. Starting clickhouse keeper with --force-recovery flag for a time
    With that flag clickhouse keeper loads configuration from raft_configuration into /keeper/config, however that presumably damages replication log in some cases in my upgrade process and leads to replication failures shortly after upgrade, I am still investigating why am I having issues with this approach.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/keeper_migration_from_23_to_24.md and reproduce the upgrade from operator 1.23.7 to 1.24.5 with three Keeper replicas. Inspect /keeper/config using the mentioned clickhouse-keeper client command and compare it with raft_configuration. Done means the migration procedure preserves valid replica addresses and replication after the topology changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
databases, distributed-systems, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.