How to override generated remote_servers

Hi Team,

I'm encountering difficulty overriding the remote_servers configuration generated by the ClickHouse Operator (CHOP) for specific pods.

Context:

I have a setup where some ClickHouse pods manage distributed tables (and need the default CHOP cluster definitions), while others do not and should not have visibility of the full cluster topology via remote_servers.
I'm attempting to apply a custom configuration selectively at the cluster level.

Attempted Solution:

Using spec.clusters.files, I added a custom file named conf.d/zzz_remote_servers_override.xml to the specific CHI cluster definition. (Using conf.d as cluster-level files cannot target config.d, and zzz_ prefix to attempt influencing load order).

This file uses <remote_servers replace="1"> and also replace="1" on nested cluster definitions (all-replicated, all-clusters) to try and remove the CHOP-generated entries and define only necessary ones (like a minimal test_cluster pointing to localhost).

Configuration File (conf.d/zzz_remote_servers_override.xml):

<clickhouse>
    <remote_servers replace="1">
        <all-replicated replace="1">
        </all-replicated>
        <all-clusters replace="1">
            <shard>
                <replica>
                    <host>127.0.0.1</host>
                    <port>9000</port>
                </replica>
            </shard>
        </all-clusters>
        <all-sharded replace="1">  <shard>
                <replica>
                    <host>127.0.0.1</host>
                    <port>9000</port>
                </replica>
            </shard>
        </all-sharded>
        <test_cluster>
            <shard>
                <replica>
                    <host>127.0.0.1</host>
                    <port>9000</port>
                </replica>
            </shard>
        </test_cluster>
    </remote_servers>
</clickhouse>

Observed Behavior:

The /var/lib/clickhouse/preprocessed/config.xml shows my conf.d/zzz_remote_servers_override.xml file is being loaded.

The test_cluster defined in my override file is present in the final merged configuration.

However, the CHOP-generated clusters (defined in config.d/chop-generated-remote_servers.xml) are also still present, indicating that replace="1" in my conf.d file did not remove them.

The preprocessed config header lists files used for generation:

<!-- This file was generated automatically.
     Do not edit it: it is likely to be discarded and generated again before it's read next time.
     Files used to generate this file:
       /etc/clickhouse-server/config.xml
       /etc/clickhouse-server/conf.d/chop-generated-hostname-ports.xml
       /etc/clickhouse-server/conf.d/chop-generated-macros.xml
       /etc/clickhouse-server/conf.d/chop-generated-zookeeper.xml
       /etc/clickhouse-server/conf.d/zzz-remote_servers_override.xml
       /etc/clickhouse-server/config.d/02-clickhouse-01-listen.xml
       /etc/clickhouse-server/config.d/02-clickhouse-02-logger.xml
       /etc/clickhouse-server/config.d/02-clickhouse-03-system_logs.xml
       /etc/clickhouse-server/config.d/chop-generated-remote_servers.xml
       /etc/clickhouse-server/config.d/chop-generated-settings.xml      -->

Questions:

Does the order of files listed in the preprocessed config header reflect the actual merge order?

Given that ClickHouse typically merges conf.d files over config.d files (as per documentation like this Altinity doc), and my conf.d file is listed, why isn't replace="1" working as expected to remove the definitions from the config.d file?

Is there a recommended or alternative way within CHOP to achieve selective overriding or removal of the auto-generated remote_servers configuration for specific pods or clusters?

Thanks for your guidance!

Start by examining the listed generated files, especially conf.d/zzz_remote_servers_override.xml and config.d/chop-generated-remote_servers.xml, and compare their order in the preprocessed config.xml. Reproduce the merge with a selectively configured cluster, then document or implement a supported CHOP approach whose final configuration omits unwanted generated remote_servers entries.