Altinity / Altinity/clickhouse-operator
CHK StatefulSets cannot change storage classes due to missing volume claims
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
CHK creates StatefulSets differently from CHI, making it very difficult to change storage classes. Unlike the ClickHouse (CHI) operator which allows for manual intervention through deletion of the PVC and POD, allowing the operator to reconcile, CHK STS store only the PVC name and not the full volume claim, which appears to lock the keeper pods on restart into a pending state.
CHK Reproduction
- Create CHK with storageClassName: ssd
- Update CHK to storageClassName: ssd2
- Delete PVC and pod to force recreation:
- Result: Pod stuck in Pending. PVC never recreated.
CHI Reproduction
- Create CHI with storageClassName: ssd
- Update CHI to storageClassName: ssd2
- Delete PVC and pod to force recreation:
- Result: Pod starts up and new PVC is created
Current CHK work around
- Delete the clickhouse-operator deployment
- Deploy the storage class update
- Locally, create new yaml files for all PVCs you will need to be replacing
- Delete the PVC of the pod you want to flip over
- Once the PVC is deleted (after the STS rolls the pod) create the new PVC from the local .yaml file (apply -f $FILE in kubectl)
- Repeat steps 4-6 for all other pods
- Redeploy the operator, which should do nothing
Ideally we'd like some way to configure the operator to recreate the PVC when a storage class is updated, but at the very least it would be good to have similar behaviour to CHI which allows for easier intervention.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the CHK StatefulSet and PVC reconciliation behavior, then compare it with the CHI behavior described in the reproduction. Reproduce the storageClassName change from ssd to ssd2, delete the PVC and pod, and trace why the PVC is not recreated. Done means CHK can recover with the updated storage class without the manual operator workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- databases, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100