[BUG] Migrate command for certain keys blocking Primary, and causing Failover.

Describe the bug

During re-shard operation, we are running Migrate command for all keys in hash slots. For some particular keys, the migrate command times out. As migrate is a blocking command, it blocks the primary and Redis thinks the primary is down, and starts a failover.

This is causing us cache availability drops, as we don't want Redis to do failover.

We tried reducing the timeout to 3 seconds, but still we get the same behaviour.

Code:

await db.KeyMigrateAsync(key, endpoint, timeoutMilliseconds: 3000);

Even though we are explicitly setting the timeout to 3 seconds, it takes the 4 seconds timeout from ConnectionMultiplexer.

StackExchange Error Logs:

TimeStamp: 2024-03-08T16:53:32.534223Z
Timeout awaiting response (outbound=0KiB, inbound=0KiB, 4100ms elapsed, timeout is 4000ms), command=MIGRATE, next: some_random_key, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, last-in: 2, cur-in: 0, sync-ops: 0, async-ops: 2193844, serverEndpoint: 172.20.0.6:6380, conn-sec: 670.01, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: mtcache000002(SE.Redis-v2.6.116.40240), PerfCounterHelperkeyHashSlot: 9271, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=8,Max=32767), POOL: (Threads=6,QueuedItems=0,CompletedItems=6635139), v: 2.6.116.40240 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts),

We get two of such errors before Redis failover, in redis, the node_timeout is set to 5 seconds.

Redis Logs:

16:53:35.041 * FAIL message received from 1a52537ed371931ec4436e02afdaae61fd061c17 about 42b37d2039622543514545a6cba3807e4db0b776
16:53:35.133 # Start of election delayed for 805 milliseconds (rank #0, offset 1269560724769).

The timestamp at which the migrate is timing out and Redis deciding to failover explains that the migrate timeout is causing this.

Update:

The first key which is blocking is of size 300 MB and has 2.3 Million keys. It is an hashset.

So what is the recommend way to migrate a hashset?

Start at the KeyMigrateAsync call and inspect how its timeout is passed through ConnectionMultiplexer, using the reported 300 MB hashset case as the reproduction. Compare the observed 4-second timeout with the requested 3 seconds and the Redis failover timing. Done should include a verified explanation or fix for the timeout behavior and guidance for migrating large hashsets.