DNSResolver: Cannot resolve host (clickhouse-keeper-0.), error 0: Host not found.

error logs:

+++ hostname -d
hostname: clickhouse-keeper-0: Address not available
++ export DOMAIN=
++ DOMAIN=
++ export CLIENT_HOST=clickhouse-keeper
++ CLIENT_HOST=clickhouse-keeper
++ export CLIENT_PORT=2181
++ CLIENT_PORT=2181
++ export RAFT_PORT=9444
++ RAFT_PORT=9444
+ source /conf/keeperFunctions.sh
++ set -ex
++ hostname -s
+ HOST=clickhouse-keeper-0
+ [[ clickhouse-keeper-0 =~ (.*)-([0-9]+)$ ]]
+ NAME=clickhouse-keeper
+ ORD=0
+ export MY_ID=1
+ MY_ID=1
+ set +e
+ getent hosts
+ [[ 0 -eq 0 ]]
+ ACTIVE_ENSEMBLE=true
+ set -e
+ mkdir -p /tmp/clickhouse-keeper/config.d/
+ [[ true == \t\r\u\e ]]
++ clickhouse-keeper-client --history-file=/dev/null -h clickhouse-keeper -p 2181 -q 'get '\''/keeper/config'\'''
Coordination::Exception: All connection tries failed while connecting to ZooKeeper. nodes: [240b:c0e0:104:544d:b464:2:0:ac6f]:2181
Poco::Exception. Code: 1000, e.code() = 113, Net Exception: No route to host (version 25.5.1.2782 (official build)), [240b:c0e0:104:544d:b464:2:0:ac6f]:2181
Poco::Exception. Code: 1000, e.code() = 0, Timeout: connect timed out: [240b:c0e0:104:544d:b464:2:0:ac6f]:2181 (version 25.5.1.2782 (official build)), [240b:c0e0:104:544d:b464:2:0:ac6f]:2181
Poco::Exception. Code: 1000, e.code() = 113, Net Exception: No route to host (version 25.5.1.2782 (official build)), [240b:c0e0:104:544d:b464:2:0:ac6f]:2181

++ true
+ CURRENT_KEEPER_CONFIG=
+ echo '<yandex><keeper_server>'
+ echo '<server_id>1</server_id>'
+ echo '<raft_configuration>'
++ echo ''
++ grep -c clickhouse-keeper-0.
+ [[ 0 == 0 ]]
+ echo '<server><id>1</id><hostname>clickhouse-keeper-0.</hostname><port>9444</port><priority>1</priority><start_as_follower>true</start_as_follower></server>'
+ IFS=
+ read -r line
++ echo ''
++ cut -d = -f 1
++ cut -d . -f 2
+ id=
+ [[ '' != '' ]]
+ IFS=
+ read -r line
+ echo '</raft_configuration>'
+ echo '</keeper_server></yandex>'
+ cat /tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml
<yandex><keeper_server>
<server_id>1</server_id>
<raft_configuration>
<server><id>1</id><hostname>clickhouse-keeper-0.</hostname><port>9444</port><priority>1</priority><start_as_follower>true</start_as_follower></server>
</raft_configuration>
</keeper_server></yandex>
+ rm -rfv /var/lib/clickhouse-keeper/terminated
+ clickhouse-keeper --config-file=/etc/clickhouse-keeper/keeper_config.xml
Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
Including configuration file '/tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml'.
Couldn't save preprocessed config to /etc/clickhouse-keeper/keeper_config-preprocessed.xml: File is read-only: /etc/clickhouse-keeper/keeper_config-preprocessed.xml
2025.05.28 05:46:16.084251 [ 23 ] {} <Information> SentryWriter: Sending crash reports is disabled
2025.05.28 05:46:16.139917 [ 23 ] {} <Information> Application: Starting ClickHouse Keeper 25.5.1.2782 (revision: 54498, git hash: e0a8a231c68824b3211369d94a592c1baab566ab, build id: 99626FE4D25A62F9C1B55F04532B0FE6DC54807B), PID 23
2025.05.28 05:46:16.139979 [ 23 ] {} <Information> Application: starting up
2025.05.28 05:46:16.140011 [ 23 ] {} <Information> Application: OS Name = Linux, OS Version = 3.10.0-1160.24.1.el7.x86_64, OS Architecture = x86_64
2025.05.28 05:46:16.140190 [ 23 ] {} <Information> Jemalloc: Value for background_thread set to true (from true)
2025.05.28 05:46:16.140390 [ 23 ] {} <Information> Application: keeper_server.max_memory_usage_soft_limit is set to 3.60 GiB
2025.05.28 05:46:16.140555 [ 23 ] {} <Information> CgroupsReader: Will create cgroup reader from '/sys/fs/cgroup/memory' (cgroups version: v1)
2025.05.28 05:46:16.140601 [ 23 ] {} <Information> MemoryWorker: Starting background memory thread with period of 50ms, using Cgroups as source
2025.05.28 05:46:16.140766 [ 23 ] {} <Debug> Application: Initializing DateLUT.
2025.05.28 05:46:16.140801 [ 23 ] {} <Trace> Application: Initialized DateLUT with time zone 'UTC'.
2025.05.28 05:46:16.146363 [ 23 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/thermal
2025.05.28 05:46:16.146417 [ 23 ] {} <Trace> AsynchronousMetrics: Scanning /sys/block
2025.05.28 05:46:16.147332 [ 23 ] {} <Trace> AsynchronousMetrics: Scanning /sys/devices/system/edac
2025.05.28 05:46:16.147556 [ 23 ] {} <Trace> AsynchronousMetrics: Scanning /sys/class/hwmon
2025.05.28 05:46:16.147837 [ 23 ] {} <Information> Context: Cannot connect to ZooKeeper (or Keeper) before internal Keeper start, will wait for Keeper synchronously
2025.05.28 05:46:16.147883 [ 23 ] {} <Debug> KeeperDispatcher: Initializing storage dispatcher
2025.05.28 05:46:16.148035 [ 23 ] {} <Information> KeeperContext: Keeper feature flag FILTERED_LIST: enabled
2025.05.28 05:46:16.148058 [ 23 ] {} <Information> KeeperContext: Keeper feature flag MULTI_READ: enabled
2025.05.28 05:46:16.148078 [ 23 ] {} <Information> KeeperContext: Keeper feature flag CHECK_NOT_EXISTS: disabled
2025.05.28 05:46:16.148098 [ 23 ] {} <Information> KeeperContext: Keeper feature flag CREATE_IF_NOT_EXISTS: disabled
2025.05.28 05:46:16.148119 [ 23 ] {} <Information> KeeperContext: Keeper feature flag REMOVE_RECURSIVE: disabled
2025.05.28 05:46:16.150470 [ 23 ] {} <Trace> KeeperSnapshotManager: Reading from disk LocalSnapshotDisk
2025.05.28 05:46:16.150584 [ 23 ] {} <Trace> KeeperSnapshotManager: No snapshots were found on LocalSnapshotDisk
2025.05.28 05:46:16.152265 [ 23 ] {} <Warning> DNSResolver: Cannot resolve host (clickhouse-keeper-0.), error 0: Host not found.
2025.05.28 05:46:16.152685 [ 23 ] {} <Error> bool DB::(anonymous namespace)::isLocalhost(const std::string &): Code: 198. DB::NetException: Not found address of host: clickhouse-keeper-0. (DNS_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000bda827b
1. DB::NetException::NetException<String const&>(int, FormatStringHelperImpl<std::type_identity<String const&>::type>, String const&) @ 0x000000000bd845a4
2. DB::(anonymous namespace)::hostByName(String const&) @ 0x000000000bd860e4
3. DB::DNSResolver::getResolvedIPAdressessWithFiltering(String const&) @ 0x000000000bd83f92
4. DB::KeeperStateManager::parseServersConfiguration(Poco::Util::AbstractConfiguration const&, bool, bool) const @ 0x000000000e6586d6
5. DB::KeeperStateManager::KeeperStateManager(int, String const&, String const&, Poco::Util::AbstractConfiguration const&, std::shared_ptr<DB::KeeperContext>) @ 0x000000000e65bf4b
6. DB::KeeperServer::KeeperServer(std::shared_ptr<DB::KeeperConfigurationAndSettings> const&, Poco::Util::AbstractConfiguration const&, ConcurrentBoundedQueue<DB::KeeperResponseForSession>&, ConcurrentBoundedQueue<DB::CreateSnapshotTask>&, std::shared_ptr<DB::KeeperContext>, DB::KeeperSnapshotManagerS3&, std::function<void (unsigned long, DB::KeeperRequestForSession const&)>) @ 0x000000000e5ca250
7. DB::KeeperDispatcher::initialize(Poco::Util::AbstractConfiguration const&, bool, bool, std::shared_ptr<DB::Macros const> const&) @ 0x000000000e5b1e59
8. DB::Context::initializeKeeperDispatcher(bool) const @ 0x000000000d076512
9. DB::Keeper::main(std::vector<String, std::allocator<String>> const&) @ 0x0000000005fa761a
10. Poco::Util::Application::run() @ 0x0000000012820b46
11. DB::Keeper::run() @ 0x0000000005fa43f0
12. mainEntryClickHouseKeeper(int, char**) @ 0x0000000005fa2fc2
13. main @ 0x0000000005fa1b73
14. ? @ 0x00007fb0c8b0ed90
15. ? @ 0x00007fb0c8b0ee40
16. _start @ 0x000000000504402e
 (version 25.5.1.2782 (official build))```




YAML:

```---
# Setup Service to provide access to ClickHouse keeper for clients
apiVersion: v1
kind: Service
metadata:
  # DNS would be like clickhouse-keeper.namespace.svc
  name: clickhouse-keeper
  labels:
    app: clickhouse-keeper
spec:
  ports:
    - port: 2181
      name: client
    - port: 7000
      name: prometheus
  selector:
    app: clickhouse-keeper
    what: node
---
# Setup Headless Service for StatefulSet
apiVersion: v1
kind: Service
metadata:
  # DNS would be like clickhouse-keeper-0.clickhouse-keepers.namespace.svc
  name: clickhouse-keepers
  labels:
    app: clickhouse-keeper
spec:
  ports:
    - port: 9444
      name: raft
  clusterIP: None
  selector:
    app: clickhouse-keeper
    what: node
---
# Setup max number of unavailable pods in StatefulSet
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: clickhouse-keeper-pod-disruption-budget
spec:
  selector:
    matchLabels:
      app: clickhouse-keeper
  maxUnavailable: 1
---
# Setup ClickHouse Keeper settings
apiVersion: v1
kind: ConfigMap
metadata:
  name: clickhouse-keeper-settings
data:
  keeper_config.xml: |
    <clickhouse>
        <include_from>/tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml</include_from>
        <logger>
            <level>trace</level>
            <console>true</console>
        </logger>
        <listen_host>0.0.0.0</listen_host>
        <listen_host>::</listen_host>
        <keeper_server incl="keeper_server">
            <enable_reconfiguration>true</enable_reconfiguration>
            <path>/var/lib/clickhouse-keeper</path>
            <tcp_port>2181</tcp_port>
            <four_letter_word_white_list>*</four_letter_word_white_list>
            <coordination_settings>
                <!-- <raft_logs_level>trace</raft_logs_level> -->
                <raft_logs_level>information</raft_logs_level>
            </coordination_settings>
            <http_control>
                <port>9182</port>
                <readiness><endpoint>/ready</endpoint></readiness>
            </http_control>
        </keeper_server>
        <prometheus>
            <endpoint>/metrics</endpoint>
            <port>7000</port>
            <metrics>true</metrics>
            <events>true</events>
            <asynchronous_metrics>true</asynchronous_metrics>
            <status_info>true</status_info>
        </prometheus>
    </clickhouse>
---
# Setup ClickHouse Keeper scripts
apiVersion: v1
kind: ConfigMap
metadata:
  name: clickhouse-keeper-scripts
  labels:
    app: clickhouse-keeper
data:
  env.sh: |
    #!/usr/bin/env bash
    export DOMAIN=`hostname -d`
    export CLIENT_HOST=clickhouse-keeper
    export CLIENT_PORT=2181
    export RAFT_PORT=9444
  keeperFunctions.sh: |
    #!/usr/bin/env bash
    set -ex
    function keeperConfig() {
      echo "$HOST.$DOMAIN:$RAFT_PORT;$ROLE;$WEIGHT"
    }
    function keeperConnectionString() {
      # If the client service address is not yet available, then return localhost
      set +e
      getent hosts "${CLIENT_HOST}" 2>/dev/null 1>/dev/null
      if [[ $? -ne 0 ]]; then
        set -e
        echo "-h localhost -p ${CLIENT_PORT}"
      else
        set -e
        echo "-h ${CLIENT_HOST} -p ${CLIENT_PORT}"
      fi
    }

  keeperStart.sh: |
    #!/usr/bin/env bash
    set -ex
    source /conf/env.sh
    source /conf/keeperFunctions.sh

    HOST=`hostname -s`
    if [[ $HOST =~ (.*)-([0-9]+)$ ]]; then
      NAME=${BASH_REMATCH[1]}
      ORD=${BASH_REMATCH[2]}
    else
      echo Failed to parse name and ordinal of Pod
      exit 1
    fi
    export MY_ID=$((ORD+1))
    set +e
    getent hosts $DOMAIN
    if [[ $? -eq 0 ]]; then
      ACTIVE_ENSEMBLE=true
    else
      ACTIVE_ENSEMBLE=false
    fi
    set -e
    mkdir -p /tmp/clickhouse-keeper/config.d/
    if [[ "true" == "${ACTIVE_ENSEMBLE}" ]]; then
      # get current config from clickhouse-keeper
      CURRENT_KEEPER_CONFIG=$(clickhouse-keeper-client --history-file=/dev/null -h ${CLIENT_HOST} -p ${CLIENT_PORT} -q "get '/keeper/config'" || true)
      # generate dynamic config, add current server to xml
      {
        echo "<yandex><keeper_server>"
        echo "<server_id>${MY_ID}</server_id>"
        echo "<raft_configuration>"
        if [[ "0" == $(echo "${CURRENT_KEEPER_CONFIG}" | grep -c "${HOST}.${DOMAIN}") ]]; then
          echo "<server><id>${MY_ID}</id><hostname>${HOST}.${DOMAIN}</hostname><port>${RAFT_PORT}</port><priority>1</priority><start_as_follower>true</start_as_follower></server>"
        fi
        while IFS= read -r line; do
          id=$(echo "$line" | cut -d '=' -f 1 | cut -d '.' -f 2)
          if [[ "" != "${id}" ]]; then
            hostname=$(echo "$line" | cut -d '=' -f 2 | cut -d ';' -f 1 | cut -d ':' -f 1)
            port=$(echo "$line" | cut -d '=' -f 2 | cut -d ';' -f 1 | cut -d ':' -f 2)
            priority=$(echo "$line" | cut -d ';' -f 3)
            priority=${priority:-1}
            port=${port:-$RAFT_PORT}
            echo "<server><id>$id</id><hostname>$hostname</hostname><port>$port</port><priority>$priority</priority></server>"
          fi
        done <<< "$CURRENT_KEEPER_CONFIG"
        echo "</raft_configuration>"
        echo "</keeper_server></yandex>"
      } > /tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml
    else
      # generate dynamic config, add current server to xml
      {
        echo "<yandex><keeper_server>"
        echo "<server_id>${MY_ID}</server_id>"
        echo "<raft_configuration>"
        echo "<server><id>${MY_ID}</id><hostname>${HOST}.${DOMAIN}</hostname><port>${RAFT_PORT}</port><priority>1</priority></server>"
        echo "</raft_configuration>"
        echo "</keeper_server></yandex>"
      } > /tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml
    fi

    # run clickhouse-keeper
    cat /tmp/clickhouse-keeper/config.d/generated-keeper-settings.xml
    rm -rfv /var/lib/clickhouse-keeper/terminated
    clickhouse-keeper --config-file=/etc/clickhouse-keeper/keeper_config.xml

  keeperTeardown.sh: |
    #!/usr/bin/env bash
    set -ex
    exec > /proc/1/fd/1
    exec 2> /proc/1/fd/2
    source /conf/env.sh
    source /conf/keeperFunctions.sh
    set +e
    KEEPER_URL=$(keeperConnectionString)
    set -e
    HOST=`hostname -s`
    if [[ $HOST =~ (.*)-([0-9]+)$ ]]; then
        NAME=${BASH_REMATCH[1]}
        ORD=${BASH_REMATCH[2]}
    else
        echo Failed to parse name and ordinal of Pod
        exit 1
    fi
    export MY_ID=$((ORD+1))

    CURRENT_KEEPER_CONFIG=$(clickhouse-keeper-client --history-file=/dev/null -h localhost -p ${CLIENT_PORT} -q "get '/keeper/config'")
    CLUSTER_SIZE=$(echo -e "${CURRENT_KEEPER_CONFIG}" | grep -c -E '^server\.[0-9]+=')
    echo "CLUSTER_SIZE=$CLUSTER_SIZE, MyId=$MY_ID"
    # If CLUSTER_SIZE > 1, this server is being permanently removed from raft_configuration.
    if [[ "$CLUSTER_SIZE" -gt "1" ]]; then
      clickhouse-keeper-client --history-file=/dev/null -q "reconfig remove $MY_ID" ${KEEPER_URL}
    fi

    # Wait to remove $MY_ID from quorum
    # for (( i = 0; i < 6; i++ )); do
    #    CURRENT_KEEPER_CONFIG=$(clickhouse-keeper-client --history-file=/dev/null -h localhost -p ${CLIENT_PORT} -q "get '/keeper/config'")
    #    if [[ "0" == $(echo -e "${CURRENT_KEEPER_CONFIG}" | grep -c -E "^server.${MY_ID}=$HOST.+participant;[0-1]$") ]]; then
    #      echo "$MY_ID removed from quorum"
    #      break
    #    else
    #      echo "$MY_ID still present in quorum"
    #    fi
    #    sleep 1
    # done

    # Wait for client connections to drain. Kubernetes will wait until the configured
    # "terminationGracePeriodSeconds" before forcibly killing the container
    for (( i = 0; i < 3; i++ )); do
      CONN_COUNT=`echo $(exec 3<>/dev/tcp/127.0.0.1/2181 ; printf "cons" >&3 ; IFS=; tee <&3; exec 3<&- ;) | grep -v "^$" | grep -v "127.0.0.1" | wc -l`
      if [[ "$CONN_COUNT" -gt "0" ]]; then
        echo "$CONN_COUNT non-local connections still connected."
        sleep 1
      else
        echo "$CONN_COUNT non-local connections"
        break
      fi
    done

    touch /var/lib/clickhouse-keeper/terminated
    # Kill the primary process ourselves to circumvent the terminationGracePeriodSeconds
    ps -ef | grep clickhouse-keeper | grep -v grep | awk '{print $1}' | xargs kill


  keeperLive.sh: |
    #!/usr/bin/env bash
    set -ex
    source /conf/env.sh
    OK=$(exec 3<>/dev/tcp/127.0.0.1/${CLIENT_PORT} ; printf "ruok" >&3 ; IFS=; tee <&3; exec 3<&- ;)
    # Check to see if keeper service answers
    if [[ "$OK" == "imok" ]]; then
      exit 0
    else
      exit 1
    fi

  keeperReady.sh: |
    #!/usr/bin/env bash
    set -ex
    exec > /proc/1/fd/1
    exec 2> /proc/1/fd/2
    source /conf/env.sh
    source /conf/keeperFunctions.sh

    HOST=`hostname -s`

    # Check to see if clickhouse-keeper service answers
    set +e
    getent hosts $DOMAIN
    if [[ $? -ne 0 ]]; then
      echo "no active DNS records in service, first running pod"
      exit 0
    elif [[ -f /var/lib/clickhouse-keeper/terminated ]]; then
      echo "termination in progress"
      exit 0
    else
      set -e
      # An ensemble exists, check to see if this node is already a member.
      # Extract resource name and this members' ordinal value from pod hostname
      if [[ $HOST =~ (.*)-([0-9]+)$ ]]; then
        NAME=${BASH_REMATCH[1]}
        ORD=${BASH_REMATCH[2]}
      else
        echo "Failed to parse name and ordinal of Pod"
        exit 1
      fi
      set +e
      HTTP_READY_STATUS=$(wget -qO- http://127.0.0.1:9182/ready)
      if [[ "0" == "$?" ]]; then
        if [[ "0" != $(echo $HTTP_READY_STATUS | grep -c '"status":"ok"') ]]; then
          echo $HTTP_READY_STATUS
          exit 0
        fi
      fi
      set -e
      MY_ID=$((ORD+1))

      CURRENT_KEEPER_CONFIG=$(clickhouse-keeper-client --history-file=/dev/null -h ${CLIENT_HOST} -p ${CLIENT_PORT} -q "get '/keeper/config'" || exit 0)
      # Check to see if clickhouse-keeper for this node is a participant in raft cluster
      if [[ "1" == $(echo -e "${CURRENT_KEEPER_CONFIG}" | grep -c -E "^server.${MY_ID}=${HOST}.+participant;1$") ]]; then
        echo "clickhouse-keeper instance is available and an active participant"
        exit 0
      else
        echo "clickhouse-keeper instance is ready to add as participant with 1 weight."

        ROLE=participant
        WEIGHT=1
        KEEPER_URL=$(keeperConnectionString)
        NEW_KEEPER_CONFIG=$(keeperConfig)
        clickhouse-keeper-client --history-file=/dev/null -q "reconfig add 'server.$MY_ID=$NEW_KEEPER_CONFIG'" ${KEEPER_URL}
        exit 0
      fi
    fi

---
# Setup ClickHouse Keeper StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  # nodes would be named as clickhouse-keeper-0, clickhouse-keeper-1, clickhouse-keeper-2
  name: clickhouse-keeper
spec:
  selector:
    matchLabels:
      app: clickhouse-keeper
  serviceName: clickhouse-keepers
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  podManagementPolicy: Parallel
  template:
    metadata:
      labels:
        app: clickhouse-keeper
        what: node
      annotations:
        prometheus.io/port: '7000'
        prometheus.io/scrape: 'true'
    spec:
      # affinity:
      #   podAntiAffinity:
      #     requiredDuringSchedulingIgnoredDuringExecution:
      #       - labelSelector:
      #           matchExpressions:
      #             - key: "app"
      #               operator: In
      #               values:
      #                 - clickhouse-keeper
      #         topologyKey: "kubernetes.io/hostname"
      volumes:
        - name: clickhouse-keeper-settings
          configMap:
            name: clickhouse-keeper-settings
            items:
              - key: keeper_config.xml
                path: keeper_config.xml
        - name: clickhouse-keeper-scripts
          configMap:
            name: clickhouse-keeper-scripts
            defaultMode: 0755
      containers:
        - name: clickhouse-keeper
          imagePullPolicy: Always
          image: "clickhouse/clickhouse-keeper:latest-alpine"
          resources:
            requests:
              memory: "4Gi"
              cpu: "2"
            limits:
              memory: "4Gi"
              cpu: "2"
          volumeMounts:
            - name: clickhouse-keeper-settings
              mountPath: /etc/clickhouse-keeper/
            - name: datadir-volume
              mountPath: /var/lib/clickhouse-keeper
            - name: clickhouse-keeper-scripts
              mountPath: /conf/
          command:
            - /conf/keeperStart.sh
          lifecycle:
            preStop:
              exec:
                command:
                  - /conf/keeperTeardown.sh
          livenessProbe:
            exec:
              command:
                - /conf/keeperLive.sh
            failureThreshold: 6
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            exec:
              command:
                - /conf/keeperReady.sh
            failureThreshold: 3
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 30
          ports:
            - containerPort: 2181
              name: client
              protocol: TCP
            - containerPort: 9444
              name: quorum
              protocol: TCP
            - containerPort: 7000
              name: metrics
              protocol: TCP
      restartPolicy: Always
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 40
  # volumeClaimTemplates:
  #   - metadata:
  #       name: clickhouse-keeper-datadir-volume
  #     spec:
  #       accessModes:
  #         - ReadWriteOnce
  #       resources:
  #         requests:
  #           storage: 25Gi

  volumeClaimTemplates:
    - metadata:
        name: datadir-volume
      spec:
        storageClassName: robin
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 50Gi```




Also please suggest the production ready, compatible versions for clickhouse-server, clickhouse-keeper and operator 

Start with the clickhouse-keeper-scripts ConfigMap, especially env.sh and keeperStart.sh, then compare the generated-keeper-settings.xml in the logs with the StatefulSet and headless Service DNS names. Reproduce the Keeper startup path and trace how DOMAIN becomes clickhouse-keeper-0.; done means the configured Keeper host resolves and the service starts without the DNSResolver error.