Altinity / Altinity/clickhouse-operator
Correct way to upgrade operator from 1.23.7 to 1.24+ with multiple clickhouse keeper replicas
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Hello
I am following https://github.com/Altinity/clickhouse-operator/blob/0.24.0/docs/keeper_migration_from_23_to_24.md instruction to upgrade operator from 1.23.7 to 1.24.5
Unfortunately I had encountered an issue during the upgrade of installation with 3 replicas of clickhouse keepers:
While with old operator version all keepers located in the single statefulset which has a single service; with 1.24.5 a separate statefulset and service is created for each replica. Which basically changes the addresses of all keeper replicas.
For example in my installation for version 1.23.7 I had 3 keeper addresses:
clickhouse-keeper-logging-0.clickhouse-keeper-logging-headless
clickhouse-keeper-logging-1.clickhouse-keeper-logging-headless
clickhouse-keeper-logging-2.clickhouse-keeper-logging-headless
And after the update they changed to:
chk-clickhouse-keeper-logging-default-0-0
chk-clickhouse-keeper-logging-default-1-0
chk-clickhouse-keeper-logging-default-2-0
I saw that <raft_configuration> was indeed changed to a correct one after an update:
<raft_configuration>
<server>
<id>0</id>
<hostname>chk-clickhouse-keeper-logging-default-0-0</hostname>
<port>9444</port>
</server>
<server>
<id>1</id>
<hostname>chk-clickhouse-keeper-logging-default-0-1</hostname>
<port>9444</port>
</server>
<server>
<id>2</id>
<hostname>chk-clickhouse-keeper-logging-default-0-2</hostname>
<port>9444</port>
</server>
</raft_configuration>
However even having the updated raft_configuration, keeper replicas were unable to reach each other, as hosts stored in replication log were not updated in the process and replicas continued using old addresses
clickhouse-keeper-logging-1:/$ clickhouse-keeper client -h localhost --port 2181 -q "get '/keeper/config'"
server.0=clickhouse-keeper-logging-0.clickhouse-keeper-logging-headless:9444;participant;1
server.2=clickhouse-keeper-logging-2.clickhouse-keeper-logging-headless:9444;participant;1
server.1=clickhouse-keeper-logging-1.clickhouse-keeper-logging-headless:9444;participant;1
I found no easy way to make clickhouse keeper reload host configuration from disk or similar issues, but maybe I am missing something here?
What I tried:
-
Configuring the cluster for keeper hosts to have the same addresses as before
I failed to come to any solution without keeping and maintaining additional k8s objects which is not desirable, as configuration had changed and we now have 3 services for 3 keeper replicas instead of one. -
Adding new hosts using zookeeper reconfigure command
Doesnt seem possible to incrementally change/keeper/configto the state matchingraft_configuration, as we can not reuse server ids -
Starting clickhouse keeper with
--force-recoveryflag for a time
With that flag clickhouse keeper loads configuration fromraft_configurationinto/keeper/config, however that presumably damages replication log in some cases in my upgrade process and leads to replication failures shortly after upgrade, I am still investigating why am I having issues with this approach.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/keeper_migration_from_23_to_24.md and reproduce the upgrade from operator 1.23.7 to 1.24.5 with three Keeper replicas. Inspect /keeper/config using the mentioned clickhouse-keeper client command and compare it with raft_configuration. Done means the migration procedure preserves valid replica addresses and replication after the topology changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- databases, distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100