Altinity / Altinity/clickhouse-operator

PVCs not deleted with "reclaimPolicy" set to "Delete"

Open
#1,695 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ongoing discussion research required
Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

I currently have a ClickHouse cluster running on AWS.
The cluster configuration was based on the following examples:

The cluster runs fine and I can create a database, etc.
When I delete the "ClickHouseInstallation" and "ClickHouseKeeperInstallation" from the cluster, I very often get PVC leftovers together with the PV and consequently the EBS volume despite setting reclaimPolicy to "Delete".

The volumeClaimTemplates section is as follows (same for both server and keeper except the name):

    volumeClaimTemplates:
      - name: ch-server-vct
        reclaimPolicy: Delete
        spec:
          storageClassName: ebs-gp3-delete
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 50Gi

The StorageClass definition for "ebs-gp3-delete" is as follows:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ebs-gp3-delete
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
  csi.storage.k8s.io/fstype: ext4
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

I haven't been able to figure out why I am getting these random leftover PVCs.
Has anyone run into a similar situation?
I would greatly appreciate any pointers that may be causing this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the server and keeper example manifests linked in the issue, then reproduce deletion of the ClickHouseInstallation and ClickHouseKeeperInstallation while observing the PVC, PV, and EBS volume lifecycle. Compare cases where reclaimPolicy is Delete with the leftover resources and identify the condition that prevents cleanup; done means the cause is documented and deletion behavior is reliably corrected or explained.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.