Single-replica shard restart updates remote_servers and leads to undesirable behaviour

In a deployment with multiple shards and a single replica each, I seem to have observed the following:

1. One of the shards has its pod restarting (in my case because Clickhouse segfaults).
2. The operator changes the remote_servers configuration to remove that server. [*]
3. Until the shard is re-added, INSERTs on distributed tables will distribute the data on shards differently (modulo the number of remaining shards), which can be undesirable when trying to enforce shard-locality of data for distributed joins. In the worst case, this can lead to local queries unexpectedly giving different results than local ones.

I have described the issue in more details there, where people suggested this was more likely an issue with the operator: https://github.com/ClickHouse/ClickHouse/issues/60219

The workaround I described there is to override the remote_servers configuration so that the shards are not removed. Instead of inserting to the "wrong" shard, the insert will fail.

[*] I have not been able to see the /etc/clickhouse-server/config.d/chop-generated-remote_servers.xml configuration change when I manually force a pod to restart by killing the process, but the fact the workaround seems to solve the issue hints that this is what happens.

Is the operator indeed removing servers with all replicas having pods in non-ready mode?
If so, it would probably be a good idea to make this behavior optional: as long as the pod exists (even if it is currently restarting), the shard should not be removed from the remote_servers.

Start by reviewing how the operator generates the remote_servers configuration and how it handles shards whose replicas are non-ready. Use the linked ClickHouse issues and the described chop-generated-remote_servers.xml change as reproduction context. Done means the behavior is confirmed and an optional way to retain existing shard entries is specified and tested.