kvcache-ai / kvcache-ai/Mooncake

[RFC]: A non-leasing existence probe (`ProbeKey` / `BatchProbeKey`)

Open
#3,769 1 comment 0 reactions 0 assignees View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

> Related PR: #3599.

## Motivation

`ExistKey` / `BatchExistKey` grant a read lease on every hit (`master_service.cpp:2780-2787` and `:2851`). This is the right behavior when an existence check means "I am about to read this object". It is the wrong behavior when the check is a speculative scan.

Speculative scans are common in KV-cache reuse: to determine how much of a cached prefix it can reuse, a client probes a set of candidate keys and then reads back only the portion it needs. The probe set is substantially larger, and issued far more frequently, than the set that is actually read — yet every probed hit currently acquires a lease.

Leases only extend, and are refreshed at half the TTL (`master_service.h:1302-1313`), so with the default 10s TTL any key re-probed within ~5s remains leased indefinitely. Eviction skips every leased object (`master_service.cpp:10250`). Together, these mean a scan-heavy client can pin effectively its entire working set.

On a small MEMORY tier with a big SSD tier, this is more than a hit-rate regression. Eviction is also the demotion path to SSD, so nothing drains, MEMORY remains full, and `Put` begins to fail — despite ample free capacity on the SSD tier behind it.

## Proposal

Provide a way to ask whether a key exists without acquiring a retention lease:

> **Point-in-time existence check.** A `true` result means the object existed at the time of the call. It may be evicted before a subsequent `Get`; callers must treat a following miss as normal.

Reads remain protected in either case — `GetReplicaList` grants its own lease (`master_service.cpp:3678`, `:3865`), so the read-in-flight window is unaffected.

There are two plausible ways to expose this, and I do not have a strong enough case to settle it unilaterally — input on the trade-off would be welcome.

### Option A — new `ProbeKey` / `BatchProbeKey` RPC

A twin of `ExistKey` / `BatchExistKey` that shares the lookup path and skips the lease.

This is purely additive: existing clients continue to call `ExistKey`, new clients opt into `ProbeKey`, and rolling upgrades remain safe in both directions. `ExistKey`'s contract is untouched, and the two operations can be documented separately with distinct guarantees.

The cost is surface area. It crosses both RPC boundaries (`WrappedMasterService` and `RealClient::*_internal`) and touches roughly 16 files including the Python binding and docs — broadly the same shape as `6105f9cd` ("[Store] add batch remove API", #1756), which serves as a reasonable template. That commit did not modify `store_c.h`, so the C ABI and the generated Rust bindings can remain untouched.

### Option B — `grant_lease` parameter on `ExistKey` / `BatchExistKey`

Considerably smaller: one API, one code path, and equally explicit at the call site.

The concern is compatibility. `coro_rpc` serializes arguments by method signature (`master_client.cpp:343`), so a server expecting an additional argument cannot decode a request from an older client, breaking mixed-version deployments during a rolling upgrade. A C++ default argument conceals this at the source level but does not address the wire format. It also changes the signature of an existing public API, including the pybind surface.

### Trade-off

Both options give callers an explicit per-call decision, and both require callers to opt in, so neither reduces client-side work. The distinguishing factor is compatibility, which favors Option A despite the larger diff.

## Relationship to #3599

#3599 adds `exist_key_grant_lease`, a master-side switch (default **on**, leaving upstream behavior unchanged) that disables probe-leasing without requiring any client change. It targets a different audience: operators running existing deployments, or third-party clients they cannot modify. A probe API serves callers whose code *can* be changed. The two appear complementary rather than competing.

Contributor guide

Open the contributing guide

Research direction

Start with the ExistKey and BatchExistKey paths in master_service.cpp, the lease logic in master_service.h, and RPC argument handling in master_client.cpp. Review related PR #3599 and compare the Option A and Option B compatibility trade-offs. Done means a settled API design with compatible client and binding surfaces, plus the necessary tests and documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend-api-design, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.