kvcache-ai / kvcache-ai/Mooncake
[CI] MasterServiceSSDSnapshotTest.EvictObject is flaky around a 4s margin
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
`MasterServiceSSDSnapshotTest.EvictObject` (`mooncake-store/tests/master_service_ssd_test_for_snapshot.cpp`) fails intermittently in the `CTest unit tests` job, passing and failing with no related change in between.
Data points so far:
- 2026-09-15T02:14, PR run [34918209366](https://github.com/kvcache-ai/Mooncake/actions/runs/34918209366) (job 104220406540): `[ FAILED ] MasterServiceSSDSnapshotTest.EvictObject (4012 ms)` — the only failure out of 200 tests.
- A main `Build & Test (Linux)` run earlier this week failed the same test at ~4011 ms (observed while triaging another PR's red).
- The same binary passes locally on a quiet machine, consistently, at ~4027 ms per run.
The signature is a test that lives right at a ~4 s margin: every observed failure sits at 4011-4012 ms, every pass just under or over the same line depending on runner load. On a shared CI runner, whatever the test waits on (eviction completing under concurrent load) crosses the boundary; on a quiet box it does not. Not a code regression — the SSD snapshot code did not change between the passing and failing runs.
Likely candidates: raise the test's own timeout/margin so runner jitter stops deciding the outcome, poll for the evicted state instead of sleeping a fixed interval, or mark it with the retry label the suite already uses for timing-sensitive cases. Filing so the flake has a home instead of a failure everyone re-runs past.
Contributor guide
Research direction
Start in mooncake-store/tests/master_service_ssd_test_for_snapshot.cpp at MasterServiceSSDSnapshotTest.EvictObject and inspect the timing-sensitive eviction wait. Run the CTest unit tests job to reproduce the failure under CI-like load, then verify the test no longer fails near the 4-second boundary and remains compatible with the suite's existing retry handling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100