TLS Connection Issue: Unable to Establish Secure Connection Between ClickHouse Operator and ClickHouse Keeper

Description

The ClickHouse operator fails to establish secure connections to ClickHouse Keeper while direct manual Keeper connection to secure port and ClickHouse-to-Keeper connections work correctly. The error suggests a TLS protocol version mismatch or incorrect TLS configuration for operator-to-keeper connections.

Error Messages

Operator Logs:

log altinity-clickhouse-operator 2025/09/15 13:32:18 connected to 100.65.20.1:9281 altinity-clickhouse-operator 2025/09/15 13:32:18 authentication failed: read tcp 100.65.25.20:42498->100.65.20.1:9281: read: connection reset by peer altinity-clickhouse-operator 2025/09/15 13:32:18 authentication failed: read tcp 100.65.25.20:53868->100.65.29.205:9281: read: connection reset by peer

Keeper Logs:

log Code: 210. DB::NetException: SSL Exception: error:0A00010B:SSL routines::wrong version number, while reading from socket

Environment

• ClickHouse Server version: 25.6.11.18
• ClickHouse Keeper version: 25.6.11.18
• ClickHouse Operator deployment: Using Helm chart v. 0.25.3

Current Setup

1. ClickHouse Keeper Configuration:

apiVersion: "clickhouse-keeper.altinity.com/v1"
kind: "ClickHouseKeeperInstallation"
metadata:
  name: keeper
spec:
  configuration:
    clusters:
      - name: "keeper-cluster"
        layout:
          replicasCount: 3
    files:
      config.xml: |        
        <clickhouse>
            <listen_host>0.0.0.0</listen_host>
            <keeper_server>
                <tcp_port_secure>9281</tcp_port_secure>
                <raft_configuration>
                  <secure>true</secure>
                  <server>
                    <id>0</id>
                    <hostname>chk-keeper-keeper-cluster-0-0.analytics.svc.cluster.local</hostname>
                    <port>9234</port>
                  </server>
                  <server>
                    <id>1</id>
                    <hostname>chk-keeper-keeper-cluster-0-1.analytics.svc.cluster.local</hostname>
                    <port>9234</port>
                  </server>
                  <server>
                    <id>2</id>
                    <hostname>chk-keeper-keeper-cluster-0-2.analytics.svc.cluster.local</hostname>
                    <port>9234</port>
                  </server>
                </raft_configuration>
            </keeper_server>
            <openSSL>
                <server>
                    <certificateFile>/etc/clickhouse-keeper/certs/tls.crt</certificateFile>
                    <privateKeyFile>/etc/clickhouse-keeper/certs/tls.key</privateKeyFile>
                    <caConfig>/etc/clickhouse-keeper/certs/selfsigned-cluster-ca-bundle.pem</caConfig>
                    <verificationMode>none</verificationMode>
                    <loadDefaultCAFile>false</loadDefaultCAFile>
                    <cacheSessions>true</cacheSessions>
                    <disableProtocols>sslv2,sslv3</disableProtocols>
                    <preferServerCiphers>true</preferServerCiphers>
                </server>
                <client>
                    <certificateFile>/etc/clickhouse-keeper/certs/tls.crt</certificateFile>
                    <privateKeyFile>/etc/clickhouse-keeper/certs/tls.key</privateKeyFile>
                    <caConfig>/etc/clickhouse-keeper/certs/selfsigned-cluster-ca-bundle.pem</caConfig>
                    <verificationMode>none</verificationMode>
                    <loadDefaultCAFile>true</loadDefaultCAFile>
                    <cacheSessions>true</cacheSessions>
                    <disableProtocols>sslv2,sslv3</disableProtocols>
                    <preferServerCiphers>true</preferServerCiphers>
                    <invalidCertificateHandler>
                        <name>RejectCertificateHandler</name>
                    </invalidCertificateHandler>
                </client>
            </openSSL>
        </clickhouse>
  defaults:
    templates:
      dataVolumeClaimTemplate: default
      podTemplate: default
  templates:
    podTemplates:
      - name: default
        spec:
          containers:
            - name: clickhouse-keeper
              volumeMounts:
                - name: clickhouse-keeper-tls-secret-volume
                  mountPath: /etc/clickhouse-keeper/certs/tls.crt
                  subPath: tls.crt
                  readOnly: true
                - name: clickhouse-keeper-tls-secret-volume
                  mountPath: /etc/clickhouse-keeper/certs/tls.key
                  subPath: tls.key
                  readOnly: true
                - name: selfsigned-cluster-ca-bundle-volume
                  mountPath: /etc/clickhouse-keeper/certs/selfsigned-cluster-ca-bundle.pem
                  subPath: selfsigned-cluster-ca-bundle.pem
                  readOnly: true
          volumes:
            - name: clickhouse-keeper-tls-secret-volume
              secret:
                secretName: clickhouse-keeper-cluster-tls-secret
            - name: selfsigned-cluster-ca-bundle-volume
              configMap:
                name: selfsigned-cluster-ca-bundle

2. ClickHouse Configuration (Working with Keeper):

    files:
      config.d/tls.xml: |
        <clickhouse>
          <!-- Disable unencrypted ports -->
          <https_port>8443</https_port>
          <tcp_port_secure>9440</tcp_port_secure>
          <listen_host>0.0.0.0</listen_host>
          <openSSL replace="replace">
            <server>
              <certificateFile>/etc/clickhouse-server/certs/tls.crt</certificateFile>
              <privateKeyFile>/etc/clickhouse-server/certs/tls.key</privateKeyFile>
              <dhParamsFile>/etc/clickhouse-server/certs/dhparam.pem</dhParamsFile>
              <verificationMode>none</verificationMode>
              <loadDefaultCAFile>false</loadDefaultCAFile>
              <caConfig>/etc/clickhouse-server/certs/selfsigned-cluster-ca-bundle.pem</caConfig> 
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
            </server>
            <client>
              <loadDefaultCAFile>true</loadDefaultCAFile>
              <caConfig>/etc/clickhouse-server/certs/selfsigned-cluster-ca-bundle.pem</caConfig>
              <certificateFile>/etc/clickhouse-server/certs/tls.crt</certificateFile>
              <privateKeyFile>/etc/clickhouse-server/certs/tls.key</privateKeyFile>
              <cacheSessions>true</cacheSessions>
              <disableProtocols>sslv2,sslv3</disableProtocols>
              <preferServerCiphers>true</preferServerCiphers>
              <verificationMode>none</verificationMode>
              <invalidCertificateHandler>
                  <name>RejectCertificateHandler</name>
              </invalidCertificateHandler>
            </client>
          </openSSL>
        </clickhouse>
    zookeeper:
      nodes:
        - host: "chk-keeper-keeper-cluster-0-0.analytics.svc.cluster.local"
          port: 9281
          secure: "yes"
        - host: "chk-keeper-keeper-cluster-0-1.analytics.svc.cluster.local"
          port: 9281
          secure: "yes"
        - host: "chk-keeper-keeper-cluster-0-2.analytics.svc.cluster.local"
          port: 9281
          secure: "yes"

3. Operator Configuration:

configs: 
  files: 
    config.yaml: 
  clickhouse: 
    access: 
      rootCA: | # CA certificates content

Current Behavior

1. ClickHouse instances successfully establish TLS connections with Keeper nodes
2. The operator fails to establish secure connections with Keeper nodes
3. Keeper logs indicate a TLS protocol version mismatch
4. Connections from the operator are being reset by Keeper

Expected Behavior

The operator should successfully establish secure TLS connections to Keeper nodes, similar to how ClickHouse instances connect.

Questions

1. Does the operator support configuring TLS protocol versions for Keeper connections?
2. Is there a way to specify detailed TLS settings for operator-to-keeper connections?
3. What is the expected configuration for operator TLS settings when connecting to a secure Keeper?
4. Are there any known limitations or requirements for operator-to-keeper TLS connections?

Additional Information

• All certificates are managed via cert-manager
• CA bundle is distributed via ConfigMap
• The same CA infrastructure is used for all components
• Direct connections between ClickHouse and Keeper work correctly
• TLS is required for all connections in the cluster
• The operator has access to the required certificates and CA bundle

No source file, test, or entry point is named. Start by locating the operator's Keeper connection and TLS configuration, then compare it with the working ClickHouse-to-Keeper settings and reproduce the reset using the supplied versions and configuration. Done means the operator establishes secure connections to all Keeper nodes and the relevant test or verification passes.