kvcache-ai / kvcache-ai/Mooncake

[Bug&RFC] Bucket LOCAL_DISK: clients sharing a `storage_path` collide on bucket ids → L3 cache broken

Open
#3,528 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

**Summary.** When multiple clients share one `storage_path` (e.g. SGLang embeds one Mooncake client per TP rank on the same `ssd_offload_path`), the bucket `LOCAL_DISK` backend has no per-client namespacing and no stable identity. This causes three coupled problems: silent collision, ambiguous recovery ownership, and per-client accounting that can neither be attributed nor survive restart. Existing issues cover pieces; none covers the **"N clients, one `storage_path`"** case.

**Environment.** Mooncake bucket `LOCAL_DISK` backend (`main`). Occurs whenever multiple clients share one `storage_path` — e.g. SGLang embeds one Mooncake client per TP rank on the same `ssd_offload_path` (sgl-project/sglang#31926).

**Three coupled problems** (`mooncake-store/src/storage_backend.cpp` unless noted):
1. **Silent collision.** `BucketIdGenerator` seeds `(time_gen() << 12) | 0` (whole-second) then only `fetch_add`s; filename `/.bucket` has no client tag; opened `O_CREAT|O_TRUNC`, no `O_EXCL`. Same-second construction → identical ids → silent overwrite. The backend struct holds only `storage_path_` — no `client_id`/pid to disambiguate.
2. **Ambiguous recovery.** `Init()` scans the whole dir and adopts every `.meta`, keyed by numeric `bucket_id`; `MasterService::AddReplica` (`master_service.cpp`) keeps one LOCAL_DISK replica per key with no `client_id` check (see #3052). Shared dir → every client claims every file → arbitrary first-claim → dangling refs when the "owner" evicts.
3. **No stable, attributable identity.** `client_id` is a random UUID minted per `Client` ctor (`client_service.cpp`, boost random), not stable across restart, and never assigned by the master. So a client cannot be recognized as the same logical owner after a restart, and its on-disk data cannot be attributed to it — which is what makes per-client, layout-independent accounting impossible today.

**Related.** #3052 (client-restart / `client_id` validation), #2306 + #2777 (marker re-adoption — marker is per-directory), #3030 (replica-id validation), #3220 + #3221 (per-object delete + GC), #2721 (SSD-capacity report), SGLang sgl-project/sglang#31926.

**The gap.** All of the above assume **one client per `storage_path`**. "N clients, one directory" is unsupported, and even the identity primitives that exist (RFC #2306's marker) are per-directory.

**Proposal** (we prefer B; A is a stopgap):
- **(A) Stopgap** — enforce one-dir-per-client (via SGLang #31926), document the contract, and fail fast on a shared directory. Leaves accounting bound to the directory layout.
- **(B) Preferred — directory-independent, Mooncake-side.** Embed a durable `owner_id` in each bucket's `.meta` (not the filename or directory). Per-client accounting and `Init()` recovery then filter by `owner_id`, so both work for **any** layout — shared dir, per-client dirs, or multiple disks. Master re-adopts by matching the in-metadata `owner_id` (not #2306's per-directory marker). Per-client bytes are reported to the master (cf. #2721); a node/disk cap is enforced by master-side aggregation, not a shared-dir scan.
- Open design point: the durable anchor by which a restarted process recovers its own `owner_id` — a master-side registry keyed by a stable client handle, or a minimal persisted id decoupled from the offload directory.

**Ask.** Decide the ownership model for a shared `storage_path`. We advocate (B) so per-client identity and accounting become layout-independent; (A)/#31926 is a stopgap. This should be coordinated with the warm re-adoption work (#2306). Willing to contribute the PR once the direction is agreed.

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Start with mooncake-store/src/storage_backend.cpp, then inspect master_service.cpp and client_service.cpp to trace bucket creation, metadata recovery, replica adoption, and client identity. Compare the proposed ownership models with related issues #3052, #2306, #2721, and #31926; done means an agreed durable ownership and accounting direction for shared storage_path, including the restart anchor.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.