We can not create connection between 2 instances after creating and close several thousands connections

Since updating our system tests to `v1.28` we're seeing some migrations are getting stuck in a `CONNECTING` state for 15m+, even though both the source and target nodes are healthy. ~50% of our test runs are hitting this issue

We see `Migration initiating` and `Connecting to target node` in a busy loop for a few seconds on the source node (logged ~30k times in 7 seconds), then no further output. Though `SLOT-MIGRATION-STATUS` returns the state is `CONNECTING`

There could be a regression on the control plane, though I don't see any related changes that could have caused this. As far as I can see the cluster configuration looks valid

Will keep looking and trying to reproduce, so will add more info...

Start by reproducing the system-test failure introduced after updating to v1.28, then inspect the source-node logs around the repeated “Migration initiating” and “Connecting to target node” messages and compare them with SLOT-MIGRATION-STATUS. Done means identifying why migrations remain in CONNECTING despite healthy nodes and preventing the busy loop or stuck state.

Written by the indexing model from the issue text.