[RFC]: Same-host Prefill Mooncake Store traffic forced over AMD AINIC RDMA and fails under load

### Changes proposed

## Environment
- **GPU**: AMD Instinct **MI355x**
- **NIC**: **AMD AINIC** (Linux RDMA devices appear as `ionic_0` … `ionic_7`)
- **Serving stack**: SGLang PD disaggregation + Mooncake Store as HiCache L3
- **Topology**: **2 Prefill on one host + 1 Decode on another host**
- Prefill P1: GPUs 0–3, AINIC `ionic_0–3`, `MOONCAKE_LOCAL_HOSTNAME=`
- Prefill P2: GPUs 4–7 (same host), AINIC `ionic_4–7`, `MOONCAKE_LOCAL_HOSTNAME=`
- Decode: remote host, Mooncake PD transfer backend
- Both Prefills share one `MOONCAKE_MASTER`, each mounts a large host-DRAM global segment (~700GB, 2MB hugepages), `MOONCAKE_PROTOCOL=rdma`, `MC_STORE_MEMCPY=1`, HiCache `write_through`
## Observed failure (from our Prefill/Decode serve logs)
Under load, Prefill-side Mooncake RDMA fails heavily on **same-host** paths (P1 ↔ P2 segments), then Decode cannot get PD KV.
**Prefill (P1/P2) symptoms**
- `cqe with error 12` on AINIC
- `transport retry counter exceeded` in Mooncake `worker_pool`
- `TRANSFER_FAIL` on HiCache keys such as `__deepseek_v4_c4`, `__deepseek_v4_c4_indexer`, `__deepseek_v4_c128`
- Endpoint instability: `Retiring endpoint ...`, `missing qp for qid 0xffffff`, `Rail paused ...`
- Transfer peers are mostly **same IP / different `ionic_*`**, e.g. `ionic_2 → @ionic_6` (cross-process, same machine)
- Only a small fraction of TE retries target the Decode host; same-host retries dominate
**Decode symptoms**
- `KVTransferError: Failed to get kvcache from prefill instance, it might be dead`
- HTTP `400` on `/v1/chat/completions`
- Decode process stays up; Prefill also continues Prefill batches (not a hard process crash)
- Decode logs little/no local `transport retry counter exceeded`; it mainly fails waiting for KV after Prefill TE degradation
Timeline pattern in our logs: same-host RDMA stress / `TRANSFER_FAIL` on Prefill first → later PD KV pull failures on Decode.
## Why we believe same-host cross-process Store access is involved
With two Prefill processes contributing segments to one Store cluster, HiCache naturally causes **process A to access objects in process B’s host segment**.
From current Store behavior:
- `MC_STORE_MEMCPY` / `LOCAL_MEMCPY` only applies when the replica endpoint matches the **same process** (including port)
- Same host alone is intentionally not enough (peer buffer VA is not valid in another process)
- Therefore P1 ↔ P2 host-segment traffic goes through **RDMA over AMD AINIC**, even though both processes are on one MI355x node
This matches the logs: most failures are same-host AINIC RDMA, not Decode-local TE errors.
## Question / ask for maintainers
1. For this deployment (multi-process Prefills on one MI355x host, shared Mooncake Store master, large host global segments), **is forcing same-host cross-process traffic onto AINIC RDMA expected?**
2. Is there an **officially supported** way today to avoid NIC hairpin for that path?
3. Would a **same-host, cross-process host-DRAM path** (e.g. share segment via shm/memfd, remap into peer VA, then local memcpy; keep cross-host PD KV on RDMA) be considered a **feasible** direction by the project? If yes, any preferred design (extend Store memcpy strategy vs TE shm transport, etc.)?
Happy to attach/sanitize the three serve logs (P1 / P2 / Decode) around the first `TRANSFER_FAIL` and Decode `KVTransferError` if useful.

### Before submitting a new issue...

- [ ] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)