Altinity / Altinity/clickhouse-operator

Liveness probe failed

Open
#1,785 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.6k
Forks
574
Avg merge
8d 6h
Merged PRs (30d)
6

Description

Keeper pods are restarting with Liveness probe failed, I am using the altinity operator for keeper and clickhouse.

I have 3 keepers

this is my yaml manifest

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: ${keeper_metadata_name}
  namespace: ${namespace}
spec:
  configuration:
    clusters:
      - name: ${keeper_cluster_name}
        layout:
          replicasCount: ${keeper_replica_count}
    settings:
      logger/level: "information"
      logger/console: "true"
      listen_host: "0.0.0.0"
      keeper_server/four_letter_word_white_list: "*"
      keeper_server/coordination_settings/raft_logs_level: "information"
      prometheus/endpoint: "/metrics"
      prometheus/port: "7000"
      prometheus/metrics: "true"
      prometheus/events: "true"
      prometheus/asynchronous_metrics: "true"
      prometheus/status_info: "false"
      keeper_server/tcp_port: 2181

  defaults:
    templates:
      # Templates are specified as default for all clusters
      podTemplate: default
      dataVolumeClaimTemplate: default

  templates:
    podTemplates:
      - name: default
        metadata:
          labels:
            app: clickhouse-keeper
        spec:
          # affinity, remove it to allow use in single node test environment
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                  - matchExpressions:
                      - key: topology.kubernetes.io/zone
                        operator: In
                        values:
%{ for zone in zones ~}
                          - ${zone}
%{ endfor ~}
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                - labelSelector:
                    matchExpressions:
                      - key: "app"
                        operator: In
                        values:
                          - clickhouse-keeper
                  topologyKey: "kubernetes.io/zone"
          containers:
            - name: clickhouse-keeper
              imagePullPolicy: IfNotPresent
              image: ${keeper_image_name}
              resources:
                requests:
                  memory: "256M"
                  cpu: "1"
                limits:
                  memory: "4Gi"
                  cpu: "2"
          securityContext:
            fsGroup: 101

          nodeSelector:
            role: "${label}"

    volumeClaimTemplates:
      - name: default
        reclaimPolicy: Retain
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: ${keeper_disk_size}
          storageClassName: ${storage_class_keeper}

failing with error

Events:                                                                                                                                                                                                                      │
│   Type     Reason     Age   From               Message                                                                                                                                                                       │
│   ----     ------     ----  ----               -------                                                                                                                                                                       │
│   Normal   Scheduled  62s   default-scheduler  Successfully assigned clickhouse/chk-agents-primary-0-0-0 to aks-chiagentpool-25729127-vmss000001                                                                             │
│   Normal   Pulled     55s   kubelet            Container image "clickhouse/clickhouse-keeper:latest" already present on machine                                                                                              │
│   Normal   Created    55s   kubelet            Created container clickhouse-keeper                                                                                                                                           │
│   Normal   Started    55s   kubelet            Started container clickhouse-keeper                                                                                                                                           │
│   Warning  Unhealthy  50s   kubelet            Liveness probe failed: Thu Aug  7 05:17:23 UTC 2025                                                                                                                           │
│ + date                                                                                                                                                                                                                       │
│ ++ exec                                                                                                                                                                                                                      │
│ bash: connect: Connection refused                                                                                                                                                                                            │
│ bash: line 1: /dev/tcp/127.0.0.1/2181: Connection refused                                                                                                                                                                    │
│ ++ printf ruok                                                                                                                                                                                                               │
│ bash: line 1: 3: Bad file descriptor                                                                                                                                                                                         │
│ ++ IFS=                                                                                                                                                                                                                      │
│ ++ tee                                                                                                                                                                                                                       │
│ bash: line 1: 3: Bad file descriptor                                                                                                                                                                                         │
│ ++ exec                                                                                                                                                                                                                      │
│ + OK=                                                                                                                                                                                                                        │
│ + [[ '' == \i\m\o\k ]]                                                                                                                                                                                                       │
│ + exit 1                                                                                                                                                                                                                     │
│   Warning  Unhealthy  45s  kubelet  Liveness probe failed: Thu Aug  7 05:17:28 UTC 2025                                                                                                                                      │
│ + date                                                                                                                                                                                                                       │
│ ++ exec                                                                                                                                                                                                                      │
│ bash: connect: Connection refused                                                                                                                                                                                            │
│ bash: line 1: /dev/tcp/127.0.0.1/2181: Connection refused                                                                                                                                                                    │
│ ++ printf ruok                                                                                                                                                                                                               │
│ bash: line 1: 3: Bad file descriptor                                                                                                                                                                                         │
│ ++ IFS=                                                                                                                                                                                                                      │
│ ++ tee

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied ClickHouseKeeperInstallation YAML and the pod Events output, focusing on the liveness probe's connection to 127.0.0.1:2181 and the Keeper startup state. Check the operator and Keeper logs to determine why the probe cannot connect; done means documenting a confirmed cause and a reproducible configuration or operational fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, yaml
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.