Altinity / Altinity/clickhouse-operator

PVCs are deleted on CHI deletion despite reclaimPolicy: Retain

Open
#1,683 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

When deleting a ClickHouseInstallation (CHI), the PersistentVolumeClaims (PVCs) associated with the ClickHouse data are being deleted—even though the reclaimPolicy: Retain is set under .spec.defaults.storageManagement and .spec.reconciling.cleanup.

We expected PVCs to persist after CHI deletion based on the operator configuration.

To Reproduce

Steps to reproduce the behavior:

Deploy ClickHouse with the following CHI manifest (simplified):

apiVersion: v1
items:
- apiVersion: clickhouse.altinity.com/v1
  kind: ClickHouseInstallationTemplate
  metadata:
    annotations:
      meta.helm.sh/release-name: clickhouse
      meta.helm.sh/release-namespace: clickhouse
    creationTimestamp: "2025-04-16T21:17:48Z"
    generation: 1
    labels:
      app.kubernetes.io/instance: clickhouse
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: clickhouse
      app.kubernetes.io/version: 24.3.12.76
      helm.sh/chart: clickhouse-0.2.2
    name: clickhouse-data
    namespace: clickhouse
    resourceVersion: "6271149"
    uid: 984e81de-4a95-4ba8-9b88-3644a30cf9af
  spec:
    templates:
      volumeClaimTemplates:
      - name: clickhouse-data
        spec:
          accessModes:
          - ReadWriteOnce
          reclaimPolicy: Retain
          resources:
            requests:
              storage: 50Gi
          storageClassName: clickhouse
---
- apiVersion: clickhouse.altinity.com/v1
  kind: ClickHouseInstallationTemplate
  metadata:
    annotations:
      meta.helm.sh/release-name: clickhouse
      meta.helm.sh/release-namespace: clickhouse
    creationTimestamp: "2025-04-16T21:17:48Z"
    generation: 1
    labels:
      app.kubernetes.io/instance: clickhouse
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: clickhouse
      app.kubernetes.io/version: 24.3.12.76
      helm.sh/chart: clickhouse-0.2.2
    name: clickhouse-pod
    namespace: clickhouse
    resourceVersion: "6271150"
    uid: 275b9720-248a-459a-b8bc-772b7b01b402
  spec:
    templates:
      podTemplates:
      - metadata:
          annotations:
            clickhouse.backup/path: /metrics
            clickhouse.backup/port: "7171"
            clickhouse.backup/scrape: "true"
            prometheus.io/path: /metrics
            prometheus.io/port: "8888"
            prometheus.io/scrape: "true"
        name: clickhouse-pod
        podDistribution:
        - scope: ClickHouseInstallation
          type: ClickHouseAntiAffinity
        spec:
          containers:
          - image: altinity/clickhouse-server:24.3.12.76.altinitystable
            imagePullPolicy: IfNotPresent
            name: clickhouse
            resources: null
            securityContext: {}
          - args:
            - server
            env:
            - name: LOG_LEVEL
              value: debug
            - name: ALLOW_EMPTY_BACKUPS
              value: "true"
            - name: API_LISTEN
              value: 0.0.0.0:7171
            - name: API_CREATE_INTEGRATION_TABLES
              value: "true"
            - name: BACKUPS_TO_KEEP_REMOTE
              value: "14"
            - name: REMOTE_STORAGE
              value: s3
            - name: S3_ACL
              value: private
            - name: S3_REGION
              value: me-central-1
            - name: S3_BUCKET
              value: clickhouse-bucket
            - name: S3_PATH
              value: backup/shard-{shard}
            - name: S3_FORCE_PATH_STYLE
              value: "true"
            - name: S3_DEBUG
              value: "true"
            image: altinity/clickhouse-backup:latest
            imagePullPolicy: Always
            name: clickhouse-backup
            ports:
            - containerPort: 7171
              name: backup-rest
          nodeSelector:
            node.kubernetes.io/instance-type: t3.medium
          securityContext: {}
          serviceAccountName: clickhouse-develop-eks-clickhouse-sa
          topologySpreadConstraints:
          - labelSelector:
              matchLabels:
                clickhouse-keeper.altinity.com/cluster: chk-test
            maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: ScheduleAnyway
---
- apiVersion: clickhouse.altinity.com/v1
  kind: ClickHouseInstallationTemplate
  metadata:
    annotations:
      meta.helm.sh/release-name: clickhouse
      meta.helm.sh/release-namespace: clickhouse
    creationTimestamp: "2025-04-16T21:17:48Z"
    generation: 1
    labels:
      app.kubernetes.io/instance: clickhouse
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: clickhouse
      app.kubernetes.io/version: 24.3.12.76
      helm.sh/chart: clickhouse-0.2.2
    name: clickhouse-service
    namespace: clickhouse
    resourceVersion: "6271148"
    uid: ec354a78-658a-4dbc-b990-5cca017abc66
  spec:
    templates:
      serviceTemplates:
      - metadata:
          labels:
            app.kubernetes.io/instance: clickhouse
            app.kubernetes.io/managed-by: Helm
            app.kubernetes.io/name: clickhouse
            app.kubernetes.io/version: 24.3.12.76
            helm.sh/chart: clickhouse-0.2.2
        name: clickhouse-service
        spec:
          ports:
          - name: http
            port: 8123
            targetPort: 8123
          - name: tcp
            port: 9000
            targetPort: 9000
          selector:
            app.kubernetes.io/instance: clickhouse
            app.kubernetes.io/name: clickhouse
          type: ClusterIP
---
- apiVersion: clickhouse.altinity.com/v1
  kind: ClickHouseInstallationTemplate
  metadata:
    annotations:
      meta.helm.sh/release-name: clickhouse
      meta.helm.sh/release-namespace: clickhouse
    creationTimestamp: "2025-04-16T21:17:48Z"
    generation: 1
    labels:
      app.kubernetes.io/instance: clickhouse
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: clickhouse
      app.kubernetes.io/version: 24.3.12.76
      helm.sh/chart: clickhouse-0.2.2
    name: clickhouse-service-lb
    namespace: clickhouse
    resourceVersion: "6271147"
    uid: 081d6190-0db7-4526-908d-f9298021e30c
  spec:
    templates:
      serviceTemplates:
      - metadata:
          annotations:
            external-dns.alpha.kubernetes.io/cloudflare-proxied: "false"
            external-dns.alpha.kubernetes.io/hostname: backend-clickhouse.xyz.com
            service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
            service.beta.kubernetes.io/aws-load-balancer-security-groups: sg-12357425473
          labels:
            app.kubernetes.io/instance: clickhouse
            app.kubernetes.io/managed-by: Helm
            app.kubernetes.io/name: clickhouse
            app.kubernetes.io/version: 24.3.12.76
            helm.sh/chart: clickhouse-0.2.2
        name: clickhouse-service-lb
        spec:
          ports:
          - name: http
            port: 8123
            targetPort: 8123
          - name: tcp
            port: 9000
            targetPort: 9000
          selector:
            app.kubernetes.io/instance: clickhouse
            app.kubernetes.io/name: clickhouse
          type: LoadBalancer
kind: List
metadata:
  resourceVersion: ""

---
apiVersion: v1
items:
- apiVersion: clickhouse.altinity.com/v1
  kind: ClickHouseInstallation
  metadata:
    annotations:
      meta.helm.sh/release-name: clickhouse
      meta.helm.sh/release-namespace: clickhouse
    creationTimestamp: "2025-04-16T21:17:48Z"
    finalizers:
    - finalizer.clickhouseinstallation.altinity.com
    generation: 1
    labels:
      app.kubernetes.io/instance: clickhouse
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: clickhouse
      app.kubernetes.io/version: 24.3.12.76
      helm.sh/chart: clickhouse-0.2.2
    name: clickhouse
    namespace: clickhouse
    resourceVersion: "6274375"
    uid: 15997da8-b57e-4df7-b65e-0bd9d313aaa6
  spec:
    configuration:
      clusters:
      - layout:
          replicasCount: 2
          shardsCount: 1
        name: clickhouse
      users:
        admin/access_management: 1
        admin/networks/ip: 0.0.0.0/0
        admin/password:
          valueFrom:
            secretKeyRef:
              key: password
              name: clickhouse-credentials
      zookeeper:
        nodes:
        - host: keeper-clickhouse
          port: 2181
    defaults:
      storageManagement:
        provisioner: Operator
        reclaimPolicy: Retain
      templates:
        clusterServiceTemplate: clickhouse-service-lb
        dataVolumeClaimTemplate: clickhouse-data
        podTemplate: clickhouse-pod
        serviceTemplate: clickhouse-service
    reconciling:
      cleanup:
        reconcileFailedObjects:
          configMap: Retain
          pvc: Retain
          service: Retain
          statefulSet: Retain
        unknownObjects:
          configMap: Retain
          pvc: Retain
          service: Retain
          statefulSet: Retain
      configMapPropagationTimeout: 90
      policy: nowait
    restart: RollingUpdate
    useTemplates:
    - name: clickhouse-service
    - name: clickhouse-service-lb
    - name: clickhouse-pod
    - name: clickhouse-data
  status:
    chop-commit: 4299a2f
    chop-date: 2025-02-24T15:26:08
    chop-ip: 10.7.3.19
    chop-version: 0.24.4
    clusters: 1
    endpoint: clickhouse-clickhouse.clickhouse.svc.cluster.local
    fqdns:
    - chi-clickhouse-clickhouse-0-0.clickhouse.svc.cluster.local
    - chi-clickhouse-clickhouse-0-1.clickhouse.svc.cluster.local
    hosts: 2
    hostsWithTablesCreated:
    - chi-clickhouse-clickhouse-0-0.clickhouse.svc.cluster.local
    - chi-clickhouse-clickhouse-0-1.clickhouse.svc.cluster.local
    pods:
    - chi-clickhouse-clickhouse-0-0-0
    - chi-clickhouse-clickhouse-0-1-0
    shards: 1
    status: Completed
    taskID: 371f0802-6775-47e0-be99-3e60f7e8c62b
    taskIDsCompleted:
    - 371f0802-6775-47e0-be99-3e60f7e8c62b
    taskIDsStarted:
    - 689101d1-2086-43ff-a24c-4aaf841fab3d
    usedTemplates:
    - name: clickhouse-service
      useType: merge
    - name: clickhouse-service-lb
      useType: merge
    - name: clickhouse-pod
      useType: merge
    - name: clickhouse-data
      useType: merge
kind: List
metadata:
  resourceVersion: ""

Delete the CHI:

kubectl delete chi clickhouse -n clickhouse

Observe the behavior:

PVCs are deleted, even though reclaimPolicy: Retain is configured.

Expected behavior

PVCs should remain (Retain) after CHI deletion to allow potential restoration or reuse of data.

Actual behavior

All CHI PVCs managed by the operator are deleted when the CHI is deleted.Only keeper PVC remains there

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Use the supplied CHI manifest to reproduce the issue, then run kubectl delete chi clickhouse -n clickhouse and inspect which PVC cleanup path removes the data PVCs. Compare that behavior with the configured storageManagement.reclaimPolicy and reconciling.cleanup.pvc settings; done means data PVCs remain after CHI deletion while the existing cleanup behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.