Altinity / Altinity/clickhouse-operator
Liveness probe failed
Open
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.6k
- Forks
- 574
- Avg merge
- 8d 6h
- Merged PRs (30d)
- 6
Description
Keeper pods are restarting with Liveness probe failed, I am using the altinity operator for keeper and clickhouse.
I have 3 keepers
this is my yaml manifest
apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
name: ${keeper_metadata_name}
namespace: ${namespace}
spec:
configuration:
clusters:
- name: ${keeper_cluster_name}
layout:
replicasCount: ${keeper_replica_count}
settings:
logger/level: "information"
logger/console: "true"
listen_host: "0.0.0.0"
keeper_server/four_letter_word_white_list: "*"
keeper_server/coordination_settings/raft_logs_level: "information"
prometheus/endpoint: "/metrics"
prometheus/port: "7000"
prometheus/metrics: "true"
prometheus/events: "true"
prometheus/asynchronous_metrics: "true"
prometheus/status_info: "false"
keeper_server/tcp_port: 2181
defaults:
templates:
# Templates are specified as default for all clusters
podTemplate: default
dataVolumeClaimTemplate: default
templates:
podTemplates:
- name: default
metadata:
labels:
app: clickhouse-keeper
spec:
# affinity, remove it to allow use in single node test environment
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: topology.kubernetes.io/zone
operator: In
values:
%{ for zone in zones ~}
- ${zone}
%{ endfor ~}
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- clickhouse-keeper
topologyKey: "kubernetes.io/zone"
containers:
- name: clickhouse-keeper
imagePullPolicy: IfNotPresent
image: ${keeper_image_name}
resources:
requests:
memory: "256M"
cpu: "1"
limits:
memory: "4Gi"
cpu: "2"
securityContext:
fsGroup: 101
nodeSelector:
role: "${label}"
volumeClaimTemplates:
- name: default
reclaimPolicy: Retain
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: ${keeper_disk_size}
storageClassName: ${storage_class_keeper}
failing with error
Events: │
│ Type Reason Age From Message │
│ ---- ------ ---- ---- ------- │
│ Normal Scheduled 62s default-scheduler Successfully assigned clickhouse/chk-agents-primary-0-0-0 to aks-chiagentpool-25729127-vmss000001 │
│ Normal Pulled 55s kubelet Container image "clickhouse/clickhouse-keeper:latest" already present on machine │
│ Normal Created 55s kubelet Created container clickhouse-keeper │
│ Normal Started 55s kubelet Started container clickhouse-keeper │
│ Warning Unhealthy 50s kubelet Liveness probe failed: Thu Aug 7 05:17:23 UTC 2025 │
│ + date │
│ ++ exec │
│ bash: connect: Connection refused │
│ bash: line 1: /dev/tcp/127.0.0.1/2181: Connection refused │
│ ++ printf ruok │
│ bash: line 1: 3: Bad file descriptor │
│ ++ IFS= │
│ ++ tee │
│ bash: line 1: 3: Bad file descriptor │
│ ++ exec │
│ + OK= │
│ + [[ '' == \i\m\o\k ]] │
│ + exit 1 │
│ Warning Unhealthy 45s kubelet Liveness probe failed: Thu Aug 7 05:17:28 UTC 2025 │
│ + date │
│ ++ exec │
│ bash: connect: Connection refused │
│ bash: line 1: /dev/tcp/127.0.0.1/2181: Connection refused │
│ ++ printf ruok │
│ bash: line 1: 3: Bad file descriptor │
│ ++ IFS= │
│ ++ tee
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the supplied ClickHouseKeeperInstallation YAML and the pod Events output, focusing on the liveness probe's connection to 127.0.0.1:2181 and the Keeper startup state. Check the operator and Keeper logs to determine why the probe cannot connect; done means documenting a confirmed cause and a reproducible configuration or operational fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, yaml
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100