Altinity / Altinity/clickhouse-operator
The replicas are dropped when adding new disks into a existing JBOD volume
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Description
When adding a new disk to an existing JBOD storage policy, the operator incorrectly treats the missing PVC for the new volume as data loss and executes SYSTEM DROP REPLICA, which removes the replica's ZooKeeper
state (including log_ptr).
Since we set None for both replica and shard in schemaPolicy, so the operator won't try to do any recovery.
Root Cause
It seems the root cause is that the PVC reconciliation flow misclassifies a new volume as data loss:
// stsReconcileOpts, migrateTableOpts = w.hostPVCsDataVolumeMissedDetectedOptions(host)
stsReconcileOpts, migrateTableOpts = w.hostPVCsDataLossDetectedOptions(host)
Any idea why we don't use hostPVCsDataVolumeMissedDetectedOptions? Any edge case it won't handle?
Steps to Reproduce
- Deploy a ClickHouseInstallation with a JBOD storage policy containing one or more disks
- Add a new disk to the JBOD volume in the CHI spec
- Observe operator logs showing SYSTEM DROP REPLICA being executed
- Verify ZooKeeper state (log_ptr, etc.) is removed for the affected replicas
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/controller/chi/worker-reconciler-chi.go around line 822 and compare hostPVCsDataVolumeMissedDetectedOptions with hostPVCsDataLossDetectedOptions. Reproduce the JBOD disk-addition scenario from the issue and trace the PVC reconciliation decision and resulting SYSTEM DROP REPLICA. Done means adding a new disk no longer removes replica ZooKeeper state or log_ptr.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100