kvcache-ai / kvcache-ai/Mooncake
[Bug]: 0.3.12.post1 Mass client expiry triggers a full-table scan under shard write locks; entire RPC layer freezes until scan completes
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
With mooncake-store master `v0.3.12.post1`, when a client node hosting many objects shuts down, at `client_ttl` seconds after its last ping the master expires all of its clients in one sweep:
```
I0811 19:41:04.965971 master_service.cpp:7061] client_id=5063333486259842164-7836975202054854303, action=client_expired
I0811 19:41:04.966004 master_service.cpp:7061] client_id=9605647728395089542-15849331516245319852, action=client_expired
I0811 19:41:04.966007 master_service.cpp:7061] client_id=14793435909040123795-1308281298534625169, action=client_expired
```
One second before, the master was fully healthy (`role=leader, state=serving, ... Ping=34.20/34.20` in admin metrics). Right after the burst, the RPC layer stops answering: requests time out, Pings are no longer processed, CPU utilization **drops** (one core works; the rest wait). The admin metrics thread keeps printing normally throughout. The process is eventually killed by our platform supervisor as unresponsive.
This is **reproducible on demand**: every shutdown of a cache-heavy client node triggers it. With CP (context parallelism) enabled in our backend it happens essentially 100% of the time; without CP, less often (our key count is several times smaller then). On `0.3.11.post1` we operated for months without a fatal incident of this kind
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Start at the client-expiration logging path in master_service.cpp around line 7061 and trace the sweep triggered after client_ttl. Reproduce a cache-heavy client shutdown, then inspect how expiration interacts with shard write locks and RPC handling. Done means the mass expiry no longer blocks the RPC layer until the scan completes, with coverage for the reported shutdown scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100