kvcache-ai / kvcache-ai/Mooncake

[CI] MasterServiceSSDSnapshotTest.EvictObject is flaky around a 4s margin

Open
#4,136 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

`MasterServiceSSDSnapshotTest.EvictObject` (`mooncake-store/tests/master_service_ssd_test_for_snapshot.cpp`) fails intermittently in the `CTest unit tests` job, passing and failing with no related change in between.

Data points so far:

- 2026-09-15T02:14, PR run [34918209366](https://github.com/kvcache-ai/Mooncake/actions/runs/34918209366) (job 104220406540): `[ FAILED ] MasterServiceSSDSnapshotTest.EvictObject (4012 ms)` — the only failure out of 200 tests.
- A main `Build & Test (Linux)` run earlier this week failed the same test at ~4011 ms (observed while triaging another PR's red).
- The same binary passes locally on a quiet machine, consistently, at ~4027 ms per run.

The signature is a test that lives right at a ~4 s margin: every observed failure sits at 4011-4012 ms, every pass just under or over the same line depending on runner load. On a shared CI runner, whatever the test waits on (eviction completing under concurrent load) crosses the boundary; on a quiet box it does not. Not a code regression — the SSD snapshot code did not change between the passing and failing runs.

Likely candidates: raise the test's own timeout/margin so runner jitter stops deciding the outcome, poll for the evicted state instead of sleeping a fixed interval, or mark it with the retry label the suite already uses for timing-sensitive cases. Filing so the flake has a home instead of a failure everyone re-runs past.

Contributor guide

Open the contributing guide

Research direction

Start in mooncake-store/tests/master_service_ssd_test_for_snapshot.cpp at MasterServiceSSDSnapshotTest.EvictObject and inspect the timing-sensitive eviction wait. Run the CTest unit tests job to reproduce the failure under CI-like load, then verify the test no longer fails near the 4-second boundary and remains compatible with the suite's existing retry handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.