kvcache-ai / kvcache-ai/Mooncake

[CI] HighPerformanceTcpLaneDistributionTest.InterleavedPeersUseEveryConfiguredLane/1 is flaky on main

Open
#4,080 1 comment 0 reactions 1 assignee Claimed by @alogfans View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

`HighPerformanceTcpLaneDistributionTest.InterleavedPeersUseEveryConfiguredLane/1` (`GetParam() = true`, sliced read) fails intermittently in `tent-ci (cuda-off)` on main.

Recent main `Build & Test (Linux)` runs show the same test alternating pass/fail with no related change in between:

- 2026-09-13T07 run 34746572253: FAILED (`1 FAILED TEST`, CTest `WriteAndSlicedRead/HighPerformanceTcpLaneDistributionTest.InterleavedPeersUseEveryConfiguredLane/1, where GetParam() = true (9 ms)`)
- the runs immediately before and after it on main: pass

The test (mooncake-transfer-engine/tent/tests/hp_tcp_transport_test.cpp:812) sets up two loopback peers on `127.0.0.1`/`127.0.0.2` with `connections_per_peer = 4` and asserts every configured lane gets used for a 4 MiB sliced read. On a shared CI runner, that assertion is timing- and scheduling-sensitive: whether all four connections per peer actually get exercised before completion depends on transfer interleaving the test does not control.

Not a code regression I can see — main's lane code didn't change between the passing and failing runs. Likely candidates: make the assertion tolerate under-utilized lanes on contended runners, bound the check to "at least N of M lanes", or serialize the sliced-read case so utilization is deterministic. Filing so the flake has a home instead of a nightly-style failure everyone re-runs past.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.