Altinity / Altinity/clickhouse-operator

The replicas are dropped when adding new disks into a existing JBOD volume

Open
#1,946 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Description

When adding a new disk to an existing JBOD storage policy, the operator incorrectly treats the missing PVC for the new volume as data loss and executes SYSTEM DROP REPLICA, which removes the replica's ZooKeeper
state (including log_ptr).

Since we set None for both replica and shard in schemaPolicy, so the operator won't try to do any recovery.

Root Cause

It seems the root cause is that the PVC reconciliation flow misclassifies a new volume as data loss:

  // stsReconcileOpts, migrateTableOpts = w.hostPVCsDataVolumeMissedDetectedOptions(host)
  stsReconcileOpts, migrateTableOpts = w.hostPVCsDataLossDetectedOptions(host)

See https://github.com/Altinity/clickhouse-operator/blob/d342fcad47126cc01bcc2fd233ee499e0d892609/pkg/controller/chi/worker-reconciler-chi.go#L822

Any idea why we don't use hostPVCsDataVolumeMissedDetectedOptions? Any edge case it won't handle?

Steps to Reproduce

  1. Deploy a ClickHouseInstallation with a JBOD storage policy containing one or more disks
  2. Add a new disk to the JBOD volume in the CHI spec
  3. Observe operator logs showing SYSTEM DROP REPLICA being executed
  4. Verify ZooKeeper state (log_ptr, etc.) is removed for the affected replicas

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/controller/chi/worker-reconciler-chi.go around line 822 and compare hostPVCsDataVolumeMissedDetectedOptions with hostPVCsDataLossDetectedOptions. Reproduce the JBOD disk-addition scenario from the issue and trace the PVC reconciliation decision and resulting SYSTEM DROP REPLICA. Done means adding a new disk no longer removes replica ZooKeeper state or log_ptr.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.