kvcache-ai / kvcache-ai/Mooncake
[Bug&RFC] Bucket LOCAL_DISK: clients sharing a `storage_path` collide on bucket ids → L3 cache broken
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
**Summary.** When multiple clients share one `storage_path` (e.g. SGLang embeds one Mooncake client per TP rank on the same `ssd_offload_path`), the bucket `LOCAL_DISK` backend has no per-client namespacing and no stable identity. This causes three coupled problems: silent collision, ambiguous recovery ownership, and per-client accounting that can neither be attributed nor survive restart. Existing issues cover pieces; none covers the **"N clients, one `storage_path`"** case.
**Environment.** Mooncake bucket `LOCAL_DISK` backend (`main`). Occurs whenever multiple clients share one `storage_path` — e.g. SGLang embeds one Mooncake client per TP rank on the same `ssd_offload_path` (sgl-project/sglang#31926).
**Three coupled problems** (`mooncake-store/src/storage_backend.cpp` unless noted):
1. **Silent collision.** `BucketIdGenerator` seeds `(time_gen() << 12) | 0` (whole-second) then only `fetch_add`s; filename `/.bucket` has no client tag; opened `O_CREAT|O_TRUNC`, no `O_EXCL`. Same-second construction → identical ids → silent overwrite. The backend struct holds only `storage_path_` — no `client_id`/pid to disambiguate.
2. **Ambiguous recovery.** `Init()` scans the whole dir and adopts every `.meta`, keyed by numeric `bucket_id`; `MasterService::AddReplica` (`master_service.cpp`) keeps one LOCAL_DISK replica per key with no `client_id` check (see #3052). Shared dir → every client claims every file → arbitrary first-claim → dangling refs when the "owner" evicts.
3. **No stable, attributable identity.** `client_id` is a random UUID minted per `Client` ctor (`client_service.cpp`, boost random), not stable across restart, and never assigned by the master. So a client cannot be recognized as the same logical owner after a restart, and its on-disk data cannot be attributed to it — which is what makes per-client, layout-independent accounting impossible today.
**Related.** #3052 (client-restart / `client_id` validation), #2306 + #2777 (marker re-adoption — marker is per-directory), #3030 (replica-id validation), #3220 + #3221 (per-object delete + GC), #2721 (SSD-capacity report), SGLang sgl-project/sglang#31926.
**The gap.** All of the above assume **one client per `storage_path`**. "N clients, one directory" is unsupported, and even the identity primitives that exist (RFC #2306's marker) are per-directory.
**Proposal** (we prefer B; A is a stopgap):
- **(A) Stopgap** — enforce one-dir-per-client (via SGLang #31926), document the contract, and fail fast on a shared directory. Leaves accounting bound to the directory layout.
- **(B) Preferred — directory-independent, Mooncake-side.** Embed a durable `owner_id` in each bucket's `.meta` (not the filename or directory). Per-client accounting and `Init()` recovery then filter by `owner_id`, so both work for **any** layout — shared dir, per-client dirs, or multiple disks. Master re-adopts by matching the in-metadata `owner_id` (not #2306's per-directory marker). Per-client bytes are reported to the master (cf. #2721); a node/disk cap is enforced by master-side aggregation, not a shared-dir scan.
- Open design point: the durable anchor by which a restarted process recovers its own `owner_id` — a master-side registry keyed by a stable client handle, or a minimal persisted id decoupled from the offload directory.
**Ask.** Decide the ownership model for a shared `storage_path`. We advocate (B) so per-client identity and accounting become layout-independent; (A)/#31926 is a stopgap. This should be coordinated with the warm re-adoption work (#2306). Willing to contribute the PR once the direction is agreed.
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Start with mooncake-store/src/storage_backend.cpp, then inspect master_service.cpp and client_service.cpp to trace bucket creation, metadata recovery, replica adoption, and client identity. Compare the proposed ownership models with related issues #3052, #2306, #2721, and #31926; done means an agreed durable ownership and accounting direction for shared storage_path, including the restart anchor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100