[Bug]: Massive performance degradation when a replica fails in multi-node SGLang HiCache with Mooncake backend

Hello!
We are constantly facing issues with multi-node SGLang HiCache deployments using the Mooncake backend.
Different root causes lead to the same behavior: massive performance degradation across all replicas.

## What can trigger the issue

### Replica tries to allocate previously used metadata port

We start up to 14 replicas with 8 clients each.

If any replica tries to use a port that was allocated earlier but is now free, or a port that is currently allocated, the following happens:

1. Mooncake on the client that failed to allocate the requested port correctly selects another port.
2. However, all other clients start performing put / get operations against the unacquired port on the host that attempted to acquire it, constantly producing logs like:

```
E0428 09:59:43.129539 12445 transfer_metadata.cpp:1154] Failed to find location of localhost:13292
E0428 09:59:43.129846 13043 transfer_metadata_plugin.cpp:290] GET http://commodity-llm-qwen35-397b-a17b-fp8-mooncake-58vblr.api-tinkoff-ai.mlc.region.local:8080/metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A13292 http=404 body: metadata not found
E0428 09:59:43.130234 11886 worker_pool.cpp:244] Worker: Cannot make connection for endpoint: localhost:13292@mlx5_0, mark it inactive
```

We mitigated this behavior by limiting the port range in https://github.com/kvcache-ai/Mooncake/pull/2008.
We also noticed that this issue can sometimes resolve itself after approximately 15 minutes.

### Short-term RDMA failure

In our cluster metrics, we observe a momentary increase in TX Discards up to 30k and Buffer Overrun up to 5 on one of the IB connections.

At the same time, all SGLang clients produce several errors like this with fixed peer_nic:
```
E0505 16:29:11.891577 12891 worker_pool.cpp:296] Worker: Process failed for slice (opcode: 0, source_addr: 0x7f204ae9c000, length: 65536, dest_addr: 0x7f6bad588000, local_nic: mlx5_9, peer_nic: localhost:14278@mlx5_5, dest_rkey: 2099456, retry_cnt: 0): transport retry counter exceeded

E0505 16:29:11.910105 12891 transfer_metadata_plugin.cpp:961] SocketHandShakePlugin: connect()10.178.54.150:16031: Connection refused [111]

E0505 16:29:11.911362 12891 worker_pool.cpp:244] Worker: Cannot make connection for endpoint: localhost:14278@mlx5_6, mark it inactive

E0505 16:30:08.311306 13626 transfer_task.cpp:372] Failed to complete transfers after 60000 milliseconds for batch 139733003265520
```

Sometimes this failure does not trigger long-term degradation: the affected replica simply terminates, and after about a minute of degradation, everything returns to normal.

## How exactly everything fails

During any of the listed failures, all SGLang clients start producing logs like:

```
E0505 16:28:08.311199 13628 transfer_task.cpp:372] Failed to complete transfers after 60000 milliseconds for batch 140292222114192
Transfer failed for key: 9225425a53e4adc63ce56594f07801a848f485a091e78d1dfa8ea96e04daece0_1_v with error: -800

W0505 16:28:08.311417 13634 client_service.cpp:1125] lease_expired_before_data_transfer_completed key=127f0a721940725feec6e971654aa6dd8e0421cba46d2aa17a18433a15cfe305_5_v
E0505 16:28:08.311813 13634 real_client.cpp:3770] BatchGet failed for key '40bb36978db6ee7051e1d854166570d784e545f4f9a69bfb1fbd4c78bf3ad360_5_k': TRANSFER_FAIL

E0505 16:29:57.696182 13638 client_service.cpp:1705] Failed to finalize put for key 150ebb8c92561b506acbecc08d7e7f2b8887c580658e3bee9a0e70b1b1325b06_7_k: OBJECT_NOT_FOUND
E0505 16:29:57.696235 13638 client_service.cpp:1882] Operation for key b5b8bd39b8606db5da4ba5e17e7def2a20dd19f5d6092d381070104f835a2822_7_k failed: OBJECT_NOT_FOUND (OBJECT_NOT_FOUND: BatchPutEnd failed; )
[2026-05-05 16:29:57 TP7] Write page to storage: 128 pages failed.

E0506 07:34:31.795542 14256 real_client.cpp:3718] Query failed for key '6779a594949ce55c621e5a7fd9661f08a498f019f68778b49770943dd8263852_4_v': RPC_FAIL
```

And mooncake_master starts to produce logs like:
```
E0505 10:30:26.174233 201 rpc_service.cpp:1117] BatchPutRevoke failed for key[0] 'd9b810cc7d1d46ccfba8e614f36d64e3346b8c0de4b0e8b7a8570d7dd4f755bd_3_k': OBJECT_NOT_FOUND
E0505 14:52:10.662626 195 master_service.cpp:896] key=e75fec615b25ef5df84e15d7b718a5a4af102a3adf5e84660e07002cefea3f1d_7_k, error=object_not_found

ERROR \u001b[0m\u001b[0K[192] [coro_connection.hpp:279] read error: Connection reset by peer, conn_id 10081

I0505 14:52:17.509413 182 master_service.cpp:3741] client_id=6505391507972573188-9897067433713881278, action=client_expired
```

As far as we understand, this means that cache transfer is effectively dead at this point: all replicas keep trying to send/receive data from a dead endpoint, blocking other activity.

We also suspect that, because the default transfer timeout is 60 seconds [here](https://github.com/kvcache-ai/Mooncake/blob/6487c6bb62952b4a4676654b4f3006457f0cc509/mooncake-store/src/transfer_task.cpp#L334), the KV cache remains locked and cannot be evicted by SGLang. This then causes performance degradation on the SGLang side.
We will try the patch from https://github.com/kvcache-ai/Mooncake/pull/2036 and check whether it reduces the degradation.

## Start configs

All nodes, except master, are connected via InfiniBands (mlx5_0 to mlx 5_11, where 5_1, 5_2, 5_7 and 5_9 are disabled ethernet)

Our topology

From `python3 -m sglang.check_env`:

```
NVIDIA Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 NIC9 NIC10 NIC11 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX PIX PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS 4,6,8,10,12 0 N/A
GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE NODE NODE PIX NODE NODE SYS SYS SYS SYS SYS SYS 4,6,8,10,12 0 N/A
GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE NODE NODE PIX NODE SYS SYS SYS SYS SYS SYS 4,6,8,10,12 0 N/A
GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS 4,6,8,10,12 0 N/A
GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS SYS SYS PIX PIX PIX NODE NODE NODE 1,3,5,7,9,11 1 N/A
GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS SYS SYS NODE NODE NODE PIX NODE NODE 1,3,5,7,9,11 1 N/A
GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE PIX NODE 1,3,5,7,9,11 1 N/A
GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE PIX 1,3,5,7,9,11 1 N/A
NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X PIX PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS
NIC1 PIX NODE NODE NODE SYS SYS SYS SYS PIX X PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS
NIC2 PIX NODE NODE NODE SYS SYS SYS SYS PIX PIX X NODE NODE NODE SYS SYS SYS SYS SYS SYS
NIC3 NODE PIX NODE NODE SYS SYS SYS SYS NODE NODE NODE X NODE NODE SYS SYS SYS SYS SYS SYS
NIC4 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE NODE NODE X NODE SYS SYS SYS SYS SYS SYS
NIC5 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE NODE NODE X SYS SYS SYS SYS SYS SYS
NIC6 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS X PIX PIX NODE NODE NODE
NIC7 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS PIX X PIX NODE NODE NODE
NIC8 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS SYS SYS PIX PIX X NODE NODE NODE
NIC9 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS SYS SYS NODE NODE NODE X NODE NODE
NIC10 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE X NODE
NIC11 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS SYS SYS NODE NODE NODE NODE NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8
NIC9: mlx5_9
NIC10: mlx5_10
NIC11: mlx5_11
```

From `ibstat`:

```
CA 'mlx5_0'
CA type: MT4129
Number of ports: 1
Firmware version: 28.41.1000
Hardware version: 0
Node GUID: 0x58a2e103001b2576
System image GUID: 0x58a2e103001b2576
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 287
LMC: 0
SM lid: 1024
Capability mask: 0xa751e848
Port GUID: 0x58a2e103001b2576
Link layer: InfiniBand
CA 'mlx5_1'
CA type: MT4125
Number of ports: 1
Firmware version: 22.38.1002
Hardware version: 0
Node GUID: 0x58a2e10300592a0c
System image GUID: 0x58a2e10300592a0c
Port 1:
State: Down
Physical state: Disabled
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x0000000000000000
Link layer: Ethernet
CA 'mlx5_10'
CA type: MT4129
Number of ports: 1
Firmware version: 28.41.1000
Hardware version: 0
Node GUID: 0x58a2e103001a691e
System image GUID: 0x58a2e103001a691e
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 284
LMC: 0
SM lid: 1024
Capability mask: 0xa751e848
Port GUID: 0x58a2e103001a691e
Link layer: InfiniBand
CA 'mlx5_11'
CA type: MT4129
Number of ports: 1
Firmware version: 28.41.1000
Hardware version: 0
Node GUID: 0x58a2e103001b2542
System image GUID: 0x58a2e103001b2542
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 254
LMC: 0
SM lid: 1024
Capability mask: 0xa751e848
Port GUID: 0x58a2e103001b2542
Link layer: InfiniBand
CA 'mlx5_2'
CA type: MT4125
Number of ports: 1
Firmware version: 22.38.1002
Hardware version: 0
Node GUID: 0x58a2e10300592a0d
System image GUID: 0x58a2e10300592a0c
Port 1:
State: Down
Physical state: Disabled
Rate: 40
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x0000000000000000
Link layer: Ethernet
CA 'mlx5_3'
CA type: MT4129
Number of ports: 1
Firmware version: 28.41.1000
Hardware version: 0
Node GUID: 0x58a2e103001a690a
System image GUID: 0x58a2e103001a690a
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 286
LMC: 0
SM lid: 1024
Capability mask: 0xa751e848
Port GUID: 0x58a2e103001a690a
Link layer: InfiniBand
CA 'mlx5_4'
CA type: MT4129
Number of ports: 1
Firmware version: 28.41.1000
Hardware version: 0
Node GUID: 0x58a2e103001b25ae
System image GUID: 0x58a2e103001b25ae
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 239
LMC: 0
SM lid: 1024
Capability mask: 0xa751e848
Port GUID: 0x58a2e103001b25ae
Link layer: InfiniBand
CA 'mlx5_5'
CA type: MT4129
Number of ports: 1
Firmware version: 28.41.1000
Hardware version: 0
Node GUID: 0x58a2e103001b2586
System image GUID: 0x58a2e103001b2586
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 238
LMC: 0
SM lid: 1024
Capability mask: 0xa751e848
Port GUID: 0x58a2e103001b2586
Link layer: InfiniBand
CA 'mlx5_6'
CA type: MT4129
Number of ports: 1
Firmware version: 28.41.1000
Hardware version: 0
Node GUID: 0x58a2e103001b258a
System image GUID: 0x58a2e103001b258a
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 285
LMC: 0
SM lid: 1024
Capability mask: 0xa751e848
Port GUID: 0x58a2e103001b258a
Link layer: InfiniBand
CA 'mlx5_7'
CA type: MT4125
Number of ports: 1
Firmware version: 22.38.1002
Hardware version: 0
Node GUID: 0x58a2e1030059282c
System image GUID: 0x58a2e1030059282c
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x0000000000000000
Link layer: Ethernet
CA 'mlx5_8'
CA type: MT4125
Number of ports: 1
Firmware version: 22.38.1002
Hardware version: 0
Node GUID: 0x58a2e1030059282d
System image GUID: 0x58a2e1030059282c
Port 1:
State: Active
Physical state: LinkUp
Rate: 100
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x00010000
Port GUID: 0x0000000000000000
Link layer: Ethernet
CA 'mlx5_9'
CA type: MT4129
Number of ports: 1
Firmware version: 28.41.1000
Hardware version: 0
Node GUID: 0x58a2e103001b25d6
System image GUID: 0x58a2e103001b25d6
Port 1:
State: Active
Physical state: LinkUp
Rate: 400
Base lid: 457
LMC: 0
SM lid: 1024
Capability mask: 0xa751e848
Port GUID: 0x58a2e103001b25d6
Link layer: InfiniBand
```

SGLang command:
```bash
MOONCAKE_TE_META_DATA_SERVER="http://mooncake_master:8080/metadata" \
MOONCAKE_MASTER="mooncake_master:50051" \
MOONCAKE_PROTOCOL="rdma" \
MOONCAKE_GLOBAL_SEGMENT_SIZE="384gb" \
SGLANG_MOONCAKE_TRANS_THREAD="8" \
MC_LOG_LEVEL="WARNING" \
MC_TE_METRIC="1" \
MC_TRANSFER_TIMEOUT="5" \
MC_STORE_CLIENT_MAX_PORT="12363" \
MC_STORE_CLIENT_MIN_PORT="12348" \
python -m sglang.launch_server \
--enable-hierarchical-cache \
--hicache-storage-backend mooncake \
--hicache-io-backend direct \
--hicache-mem-layout page_first_direct \
--hicache-write-policy write_through \
--model /models/Qwen3.5-397B-A17B-FP8 \
--enable-metrics \
--log-level-http=warning \
--host 0.0.0.0 \
--port 8000 \
--tp=8 \
--kv-cache-dtype fp8_e4m3 \
--mamba-scheduler-strategy extra_buffer \
--mamba-full-memory-ratio=0.5 \
--page-size 64 \
--tool-call-parser=qwen3_coder \
--speculative-algorithm=EAGLE \
--speculative-num-steps=3 \
--speculative-eagle-topk=1 \
--speculative-num-draft-tokens=4 \
--enable-flashinfer-allreduce-fusion \
--mem-fraction-static=0.8 \
--model-loader-extra-config='{"enable_multithread_load":"true","num_threads":64}' \
--enable-cache-report
```

Mooncake master command:
```bash
mooncake_master --enable_http_metadata_server=true --eviction_high_watermark_ratio=0.95 --enable_metric_reporting=true
```