Schema Skew Issue

Hi team.
We are using the operator . and sometimes due to some race condition we endup with permanent schema skew.

examples: if there was some error while migrateTables() run,the error is simply logged and proceed (Host is pushed and added to HostWithTablesCreated() meaning it will not attempt to fix the schema skew. also the CHI.status transition to completed, and all shards pods mark as ready regardless of the status since they rely on /ping.

Or even if there are 'Create Table .. ON CLUSTER' running on shard-0 while shard-1 is still bootstrap it might miss some of queries since kubelet syncFrequency (by default 1m) will take time till shard-0 aware shard-1 added to the actual remote_servers.xml on Disk.

I wondering what is the best way to solve it since it seems the Operator is 'best-effort' for DDL alignment. I thought to migrate and use Replicated Database engine which solve exactly that. but even if I will switch the Engine, in case of error It seems the Operator is still best-effort for Run the Create Database replicated on new pods/shards.

for example: shard-2 join the cluster the operator will attempt to run 'Create Database Engine = Replicated'. but if also that fail it will add that Host to cluster.

I thought about, maybe not swallow the error on migrateTables(), and raise the error. and maybe re-queue so the reconcile will be retried? or there is reason having the operator working on best-effort strategy?
even if migrateTables succeed or fail we will end-up with Host added to HostWithTabelsCreated(). and also Host added to remote_servers.xml which might get us into permanent schema skew

maybe readiness gate but base on having lag related (DDL/Replication Lag) https://github.com/Altinity/clickhouse-operator/issues/1662 readiness probe will do more harm than good.

We are using versions 0.25.2 and 0.26.3

Thanks

Start by tracing migrateTables(), HostWithTablesCreated(), and the reconcile path that adds hosts to remote_servers.xml. Review how migration errors affect CHI.status, shard readiness, and retry or re-queue behavior, using the reported bootstrap and /ping scenarios as cases. Done should define and verify consistent behavior when schema or replicated-database creation fails, but the issue does not specify the intended policy.