kvcache-ai / kvcache-ai/Mooncake
[Bug] TcpTransport connection pool: unbounded growth with threshold mass-GC; stale entries to restarted peers are never evicted
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
## Environment
mooncake-transfer-engine-cuda13 0.3.11.post1 (PyPI), used as SGLang PD-disaggregation KV transfer backend (`MC_FORCE_TCP=1`, `MC_TCP_ENABLE_CONNECTION_POOL=true`), same-host prefill(TP4) → decode(TP4), 8×B300, container network.
## Issue 1: pool grows unboundedly, then mass-closes tens of thousands of connections at once
Under sustained load (fixed 10-way concurrency, ~27K-token KV parcels, chunked), `/proc/net/tcp` on the sender side shows a sawtooth with ~8-minute period:
```text
03:15 ESTABLISHED 29024
03:17 ESTABLISHED 38952
03:18 ESTABLISHED 17724 TIME_WAIT 21089 <- ~21K connections closed within one minute
03:24 ESTABLISHED 44419
03:25 ESTABLISHED 45623 <- 83% of the ephemeral port range
03:26 ESTABLISHED 20738 TIME_WAIT 26831 <- ~27K closed at once
```
Two problems: (a) between flushes the pool keeps growing toward the ephemeral-port limit (we measured peaks at 83% of a widened 10240-65000 range — with the default ~28K-port range this would hit `EADDRNOTAVAIL` long before the flush); (b) the flush itself dumps 20K+ sockets into TIME_WAIT within seconds. A bounded pool with per-peer connection reuse would presumably hold this at a few hundred connections (we see ~566 ESTABLISHED steady when idle-ish).
## Issue 2: pooled connections to a dead/restarted peer are never evicted
If one side of the PD pair is restarted alone (its listen ports change), the surviving side's pool keeps the stale entries. Small transfers still succeed (presumably via new connections), but **large multi-slice transfers stall silently and permanently** — the sender reports no error, the receiver times out (`KVPoll.WaitingForInput` timeout after 300s). It also degrades progressively: after one single-sided peer restart we only measured throughput loss at high concurrency; after a second one, all large transfers hung. Restarting both sides together fully restores normal behavior.
## Ask
1. Bound the pool (per-peer cap + LRU reuse) instead of grow-then-mass-GC.
2. Evict/reconnect pool entries on connection failure or peer-identity change, so a peer restart doesn't silently poison large transfers.
Happy to provide more traces or test patches — this is reproducible on demand in our setup.
Contributor guide
Assessment
This issue has not been assessed yet.