kvcache-ai / kvcache-ai/Mooncake

[RFC]: Decouple Mooncake Store build targets and standardize its SDK boundary

Open
#3,181 2 comments 0 reactions 0 assignees View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Changes proposed

## Summary

Mooncake Store currently mixes the Master control plane, Client/SDK data plane, Transfer Engine, portable dependencies, and language bindings in one native build boundary. This causes three recurring problems:

1. `mooncake_master` inherits client and hardware dependencies such as Transfer Engine, curl, RDMA, and CUDA.
2. Python, Go, and Rust duplicate Store's internal native dependency list.
3. Distributable Store builds depend on compatible versions of several host C/C++ libraries, including foundational logging and optional Store paths.

This RFC proposes one coordinated cleanup:

- split the Master and Client/SDK CMake target graphs;
- bundle the initial dependency set from fixed in-repository sources; and
- make one `libmooncake_store.so` the native SDK boundary consumed by language bindings through `store_c.h`.

## Why this needs an RFC

Recent changes solve parts of the problem:

- #3078 added an optional C shared library and Rust `dlopen` support;
- #3119 removed an accidental CUDA dependency from `mooncake_master`; and
- #2069 documented the Rust binding's duplicated native link requirements.

Those fixes are useful foundations, but the underlying CMake target graph still couples the Master, SDK, and implementation dependencies. This RFC proposes a single direction so follow-up PRs do not create competing shared libraries or additional compatibility layers.

## Proposed design

### 1. Separate Master from Client/SDK

`mooncake_master` should link only server/core targets. It must not depend on the Store SDK or Transfer Engine target and should not directly require curl, libibverbs, CUDA, or other client/hardware libraries.

The Client/SDK may continue to depend on Transfer Engine and hardware/system libraries. It must do so through real CMake targets and feature conditions.

With `WITH_TE=OFF, WITH_STORE=ON`:

- the Master remains buildable;
- the Client/SDK is disabled with a clear status message; and
- no bare `-ltransfer_engine` item is generated.

Affected CMake state should be target-scoped rather than directory-global.

### 2. Bundle the dependency set

Business targets consume stable project aliases instead of raw library names. The aliases always resolve to fixed in-repository sources; there is no alternative host-package mode.

The initial fixed set is xxHash 0.8.3, Zstandard 1.5.7, JsonCpp 1.9.6, yaml-cpp 0.8.0, standalone Asio 1.30.2, gflags 2.3.0, glog 0.7.1, hiredis 1.4.0, liburing 2.14, and libzmq 4.3.5. Compiled libraries are static, PIC, and hidden; Asio remains header-only. Sources, hashes, and licenses are recorded in the repository, and configure does not download them.

gflags and glog are default foundation dependencies. hiredis, liburing, and libzmq are built and linked only when their existing Redis, io_uring, or KV event features require them. Bundling must not turn optional features into unconditional default build work. Store uses libzmq's C API, so cppzmq is not added.

This RFC does not propose bundling curl, RDMA, CUDA, or every optional dependency.

### 3. Use one shared Store SDK boundary

The canonical native Store SDK should be `libmooncake_store.so`. Python, Go, and Rust should directly link only this Mooncake library rather than reproduce its implementation dependency list.

`store_c.h` remains the stable cross-language ABI. The shared library must:

- preserve the existing 19 C functions;
- absorb the bundled portable dependencies;
- hide their implementation symbols; and
- retain Rust `dlopen` support introduced by #3078.

The implementation should evolve #3078's shared-library work rather than introduce a second shared object. Existing options and target names need a compatibility mapping or a documented deprecation period.

Python currently exposes tensor/Engram behavior beyond the existing C header. That public behavior must be preserved. A complete source-level migration of those APIs to the C ABI may therefore be a follow-up, even though the Python extension's direct native library boundary can be reduced immediately.

## Scope boundaries

This RFC does not include:

- curl replacement or hardware-library pluginization;
- changes to Store data placement or wire protocols;
- a new public C++ ABI containing third-party types; or
- global use of `BUILD_SHARED_LIBS`.

The SDK may still directly require system hardware libraries when the corresponding client feature is enabled. Removing those dependencies is a later phase.

## Compatibility

- Existing Store C APIs remain unchanged.
- Existing target and artifact names are retained through aliases where practical.
- Rust's #3078 `dlopen` mode remains supported.
- TCP behavior remains unchanged.
- Any change to Master RDMA control-plane support must be explicitly agreed before implementation; it must not be removed as an incidental linker fix.

## Implementation sequence

The work should be submitted as a dependent series:

1. **Bundled dependencies:** add the ten fixed dependencies and migrate common consumers to stable aliases; keep hiredis, liburing, and libzmq feature-conditional.
2. **Store targets and SDK:** split Master/Client targets, fix `WITH_TE=OFF`, and evolve the shared artifact from #3078 into the canonical SDK.
3. **Bindings and packaging:** remove duplicated Python/Go/Rust link lists and update wheel/install/CI paths.

If steps 2 and 3 cannot independently keep the repository green, they should be combined rather than merging a broken intermediate state.

## Validation

The implementation must verify:

- the ten bundled dependencies build from checked-in sources without network access when their consumers are enabled;
- `WITH_TE=OFF, WITH_STORE=ON` builds the Master;
- Master `DT_NEEDED` contains no curl, ibverbs, CUDA, or Transfer Engine;
- binding binaries directly depend only on `libmooncake_store.so` as a Mooncake library;
- bundled dependency symbols are not exported; and
- representative C++, Python, Go, and Rust tests pass where toolchains are available.

A local prototype reduced a clean Master-only build from 251.51 seconds to 81.19 seconds and a representative incremental test compile from 34.98 seconds to 7.38 seconds. Its clean full-SDK build increased from 249.66 seconds to 311.69 seconds, so full-SDK build cost must be profiled or explicitly accepted before merge.

## Questions for consensus

1. How should the option/target introduced by #3078 transition to the canonical SDK?
2. Must Master RDMA control-plane support remain in the initial target split?

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing CMake target graph, the shared-library work from #3078, and the stable ABI in store_c.h. Trace how the Master, Client/SDK, Transfer Engine, and language bindings currently connect. Done means the proposed target split, bundled dependency behavior, canonical SDK boundary, compatibility requirements, and validation checks are implemented without breaking the listed C++, Python, Go, and Rust tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, go, python, rust
Domain
api, build-system
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.