ClickHouse retries on not-ready servers because of `PublishNotReadyAddresses`

If a server failed to start, DNS should fail to resolve (server name does not correspond to any IP address), so as to tell the initiator ClickHouse server to avoid connecting to this server.

With PublishNotReadyAddresses=true, the initiator ClickHouse server believes a failed server should exist and makes several retries to connect it. Thus when there is any failed server, all subsequent queries will slow down for several seconds to wait for the connection retries to run out.

PublishNotReadyAddresses here https://github.com/Altinity/clickhouse-operator/blob/69d5e39807e13feef00f47434d8a9fd9f4f0430f/pkg/model/chk/creator/service.go#L227

As k8s docs says:

The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery

But a ClickHouse cluster does not need peer discovery because it is designed as multi-master architecture, where peer addresses are resolved from their hostnames before query execution (perhaps with DNS cache). An IP address of failed server does not make it better.

Start in pkg/model/chk/creator/service.go around the PublishNotReadyAddresses setting referenced in the issue. Check how the service is rendered and identify the existing tests for service creation. Done means failed servers are not published through this setting and the generated service behavior no longer causes unnecessary connection retries.