Altinity / Altinity/clickhouse-operator

Operator Does not Render the Correct Keeper Statefulset Volumes for Secrets

Open
#1,694 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Keeper
Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

I have this ClickHouse Keeper installation [1], I'm seeing an issue with keeper Statefulsets failing to create keeper pods because the pod template has some volumes of type secret with no name (and those volumes are marked with optional: false). It surfaces as this event [2] on the Statefulsets, with no pods created.
I'm adding those secrets in the .spec.configuration.files of my ClickHouseKeeperInstallation just like this example, but when I look at the -o yaml of the keeper Statefulsets that's generated by the operator I see this [3].
Am I doing something wrong?

Notes:

  • I'm using operator version 0.24.5
  • This is hosted on EKS with K8s version 1.31
  • The secrets exist and are being used by the clickhouse pods in this ClickHouseInstallation [4] with no issues.

[1] ClickHouseKeeperInstallation manifest:

apiVersion: 'clickhouse-keeper.altinity.com/v1'
kind: 'ClickHouseKeeperInstallation'
metadata:
  name: 'keeper'
  namespace: 'clickhouse'
spec:
  configuration:
    settings:
      prometheus/endpoint: '/metrics'
      prometheus/port: '9363'
      prometheus/metrics: 'true'
      prometheus/events: 'true'
      prometheus/asynchronous_metrics: 'true'
      prometheus/errors: 'true'
      prometheus/status_info: 'true'
    files:
      keeper_config.xml: |
        <clickhouse>
            <keeper_server>
              <tcp_port_secure>9281</tcp_port_secure>
              <raft_configuration>
                <secure>true</secure>
              </raft_configuration>
            </keeper_server>
        </clickhouse>
      openssl_server.xml: |
        <clickhouse>
          <openSSL>
            <server>
              <certificateFile>/etc/clickhouse-server/secrets.d/tls.crt/clickhouse-certs/tls.crt</certificateFile>
              <privateKeyFile>/etc/clickhouse-server/secrets.d/tls.key/clickhouse-certs/tls.key</privateKeyFile>
              <caConfig>/etc/clickhouse-server/secrets.d/ca.crt/clickhouse-certs/ca.crt</caConfig>
              <verificationMode>relaxed</verificationMode>
              <loadDefaultCAFile>true</loadDefaultCAFile>
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
              <dhParamsFile remove="remove">/etc/clickhouse-keeper/dhparam.pem</dhParamsFile>
            </server>
          </openSSL>
        </clickhouse>
      openssl_client.xml: |
        <clickhouse>
          <openSSL>
            <client>
              <loadDefaultCAFile>true</loadDefaultCAFile>
              <caConfig>/etc/clickhouse-server/secrets.d/ca.crt/clickhouse-certs/ca.crt</caConfig>
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
              <invalidCertificateHandler>
                  <name>RejectCertificateHandler</name>
              </invalidCertificateHandler>
            </client>
          </openSSL>
        </clickhouse>
      tls.crt:
        valueFrom:
          secretKeyRef:
            name: clickhouse-certs
            key: tls.crt
      tls.key:
        valueFrom:
          secretKeyRef:
            name: clickhouse-certs
            key: tls.key
      ca.crt:
        valueFrom:
          secretKeyRef:
            name: clickhouse-certs
            key: ca.crt
    clusters:
      - name: 'main-ensemble'
        templates:
          dataVolumeClaimTemplate: data-volume-template
          logVolumeClaimTemplate: log-volume-template
        layout:
          replicas:
            - templates:
                podTemplate: clickhouse-keeper-pod-template
            - templates:
                podTemplate: clickhouse-keeper-pod-template
            - templates:
                podTemplate: clickhouse-keeper-pod-template
  templates:
    podTemplates:
      - name: clickhouse-keeper-pod-template
        metadata:
          annotations:
            prometheus.io/scrape: 'true'
        spec:
          containers:
            - name: clickhouse-keeper
              image: clickhouse/clickhouse-keeper:23.12.5.81
              ports:
                - name: metrics
                  containerPort: 9363
    volumeClaimTemplates:
      - name: data-volume-template
        spec:
          storageClassName: keeper-data
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 25Gi
      - name: log-volume-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi

[2] Event of one of the keeper statefulsets that failed to create a pod:

Events:                                                                                                                                                                                                                                 │
│   Type     Reason        Age                 From                    Message                                                                                                                                                            │
│   ----     ------        ----                ----                    -------                                                                                                                                                            │
│   Warning  FailedCreate  88s (x19 over 23m)  statefulset-controller  create Pod chk-keeper-main-ensemble-0-0-0 in StatefulSet chk-keeper-main-ensemble-0-0 failed error: Pod "chk-keeper-main-ensemble-0-0-0" is invalid: [spec.volumes │
│ 5[].secret.secretName: Required value, spec.volumes[5].secret.items[0].key: Required value, spec.volumes[5].secret.items[0].path: Required value, spec.volumes[6].secret.secretName: Required value, spec.volumes[6].secret.items[0].ke │
│ y: Required value, spec.volumes[6].secret.items[0].path: Required value, spec.volumes[7].secret.secretName: Required value, spec.volumes[7].secret.items[0].key: Required value, spec.volumes[7].secret.items[0].path: Required value,  │
│ spec.containers[0].volumeMounts[3].name: Not found: "tlscrt", spec.containers[0].volumeMounts[4].name: Not found: "tlskey", spec.containers[0].volumeMounts[5].name: Not found: "cacrt"]

[3] the volumes section of the statefulsets having no key or path even tho they were specified in the ClickHouseKeeperInstallation:

spec.template.spec.volumes:
      ...
      - name: tlskey
        secret:
          defaultMode: 420
          items:
          - key: ""
            path: ""
      - name: cacrt
        secret:
          defaultMode: 420
          items:
          - key: ""
            path: ""
      - name: tlscrt
        secret:
          defaultMode: 420
          items:
          - key: ""
            path: ""

[4] The ClickHouseInstallation manifest that is successfully using the same secrets:

apiVersion: 'clickhouse.altinity.com/v1'
kind: 'ClickHouseInstallation'
metadata:
  name: 'ch'
  namespace: clickhouse
spec:
  configuration:
    settings:
      prometheus/endpoint: '/metrics'
      prometheus/port: '9363'
      prometheus/metrics: 'true'
      prometheus/events: 'true'
      prometheus/asynchronous_metrics: 'true'
      prometheus/errors: 'true'
      prometheus/status_info: 'true'
      ...

    files:
      openssl_server.xml: |
        <clickhouse>
          <openSSL>
            <server>
              <certificateFile>/etc/clickhouse-server/secrets.d/tls.crt/clickhouse-certs/tls.crt</certificateFile>
              <privateKeyFile>/etc/clickhouse-server/secrets.d/tls.key/clickhouse-certs/tls.key</privateKeyFile>
              <caConfig>/etc/clickhouse-server/secrets.d/ca.crt/clickhouse-certs/ca.crt</caConfig>
              <verificationMode>relaxed</verificationMode>
              <loadDefaultCAFile>true</loadDefaultCAFile>
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
            </server>
          </openSSL>
        </clickhouse>
      openssl_client.xml: |
        <clickhouse>
          <openSSL>
            <client>
              <loadDefaultCAFile>true</loadDefaultCAFile>
              <caConfig>/etc/clickhouse-server/secrets.d/ca.crt/clickhouse-certs/ca.crt</caConfig>
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
              <invalidCertificateHandler>
                  <name>RejectCertificateHandler</name>
              </invalidCertificateHandler>
            </client>
          </openSSL>
        </clickhouse>
      tls.crt:
        valueFrom:
          secretKeyRef:
            name: clickhouse-certs
            key: tls.crt
      tls.key:
        valueFrom:
          secretKeyRef:
            name: clickhouse-certs
            key: tls.key
      ca.crt:
        valueFrom:
          secretKeyRef:
            name: clickhouse-certs
            key: ca.crt
    zookeeper:
      nodes:
        - host: chk-keeper-main-ensemble-0-0
          port: 9281
          secure: 'yes'
        - host: chk-keeper-main-ensemble-0-1
          port: 9281
          secure: 'yes'
        - host: chk-keeper-main-ensemble-0-2
          port: 9281
          secure: 'yes'
    clusters:
      - name: 'main-cluster'
        secure: 'yes'
        insecure: 'no'
        secret:
          auto: 'yes'
        templates:
          dataVolumeClaimTemplate: data-volume-template
          logVolumeClaimTemplate: log-volume-template
          podTemplate: clickhouse-pod-template
        layout:
          shardsCount: 3
          replicas: 2
  templates:
    podTemplates:
      - name: clickhouse-pod-template
        metadata:
          annotations:
            prometheus.io/scrape: 'true'
        spec:
          containers:
            - name: clickhouse
              image: clickhouse/clickhouse-server:23.12.5.81
              ports:
                - name: metrics
                  containerPort: 9363
            - name: clickhouse-backup
              image: altinity/clickhouse-backup:stable
              imagePullPolicy: IfNotPresent
              ...
    volumeClaimTemplates:
      - name: data-volume-template
        spec:
          storageClassName: ch-sc
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 25Gi
      - name: log-volume-template
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 5Gi

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by applying the ClickHouseKeeperInstallation manifest and inspecting the generated Keeper StatefulSet with kubectl -o yaml, then compare its secret volumes with the configuration.files entries. Reproduce the FailedCreate event and trace the operator path that renders those volumes; done means the generated StatefulSet contains each secret name, key, and path and its pods are created.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.