Altinity / Altinity/clickhouse-operator

CHK StatefulSets cannot change storage classes due to missing volume claims

Open
#1,808 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Keeper
Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

CHK creates StatefulSets differently from CHI, making it very difficult to change storage classes. Unlike the ClickHouse (CHI) operator which allows for manual intervention through deletion of the PVC and POD, allowing the operator to reconcile, CHK STS store only the PVC name and not the full volume claim, which appears to lock the keeper pods on restart into a pending state.

CHK Reproduction

  1. Create CHK with storageClassName: ssd
  2. Update CHK to storageClassName: ssd2
  3. Delete PVC and pod to force recreation:
  4. Result: Pod stuck in Pending. PVC never recreated.

CHI Reproduction

  1. Create CHI with storageClassName: ssd
  2. Update CHI to storageClassName: ssd2
  3. Delete PVC and pod to force recreation:
  4. Result: Pod starts up and new PVC is created

Current CHK work around

  1. Delete the clickhouse-operator deployment
  2. Deploy the storage class update
  3. Locally, create new yaml files for all PVCs you will need to be replacing
  4. Delete the PVC of the pod you want to flip over
  5. Once the PVC is deleted (after the STS rolls the pod) create the new PVC from the local .yaml file (apply -f $FILE in kubectl)
  6. Repeat steps 4-6 for all other pods
  7. Redeploy the operator, which should do nothing

Ideally we'd like some way to configure the operator to recreate the PVC when a storage class is updated, but at the very least it would be good to have similar behaviour to CHI which allows for easier intervention.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the CHK StatefulSet and PVC reconciliation behavior, then compare it with the CHI behavior described in the reproduction. Reproduce the storageClassName change from ssd to ssd2, delete the PVC and pod, and trace why the PVC is not recreated. Done means CHK can recover with the updated storage class without the manual operator workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.