kvcache-ai / kvcache-ai/Mooncake

[Bug]: Silent offload failure: keys exist on offload SSD but return empty value after eviction offload

Open
#3,465 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Bug Report

### Describe the bug
We observe silent offload failure against local SSD offload backend. Keys are visible via `/get_all_keys`, but corresponding values return empty when fetched.

We are testing with recent Mooncake commit: https://github.com/kvcache-ai/Mooncake/commit/67aaf6de74f35c7c5cfd072e8643b0a80eb5999a (2026‑08‑12).
The same workload with June‑end Mooncake code works perfectly without empty‑value issue. This regression appears after we upgraded to newer codebase.

### Master launch command:

```C++
mooncake_master \
--rpc_port=8001 \
--rpc_thread_num=8 \
--metrics_port=8007 \
--default_kv_lease_ttl=5000 \
--allow_evict_soft_pinned_objects=true \
--eviction_ratio=0.35 \
--eviction_high_watermark_ratio=0.7 \
--cluster_id=mooncake_cluster \
--allocation_strategy=free_ratio_first \
--quota_bytes=3298534883328 \
--enable_ha=false \
--enable_offload=true \
--offload_on_evict=true \
--offload_force_evict=false \
--enable_disk_eviction=true \
--promotion_on_hit=false \
--put_start_discard_timeout_sec=300 \
--put_start_release_timeout_sec=600

```
### Real‑Client Environment Variables & Settings
``` C++
MC_IB_PCI_RELAXED_ORDERING = 1
MOONCAKE_LOCAL_BUFFER_SIZE = 6442450944
MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY = lru
MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT = 128
MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE = 3113851289600
MOONCAKE_OFFLOAD_BUCKET_SIZE_LIMIT_BYTES = 268435456
MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_INTERVAL_SECONDS = 2
MOONCAKE_OFFLOAD_FILE_STORAGE_PATH = /ssd_file/mooncake_cache/
MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS = 1
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES = 21474836480
MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR = bucket_storage_backend
MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES = 3221225472000
MOONCAKE_OFFLOAD_USE_URING = 1

```
Client launch command:
```C++
mooncake_client \
--master_server_address=:8001 \
--host= \
--protocol=rdma \
--port=8005 \
--global_segment_size=25GB \
--enable_offload=true \
--threads=8 \
--metadata_server=P2PHANDSHAKE
```
## To Reproduce

1. Start master and real‑client with above configurations, `offload_on_evict=true`.
2. Write ~100GB total data. Logs confirm all keys‑values are successfully written into memory.
3. With `offload_on_evict=true`, objects should be offloaded to local SSD and persist. No heavy SSD‑side eviction is expected under this workload.
4. After writing completes, read back keys.
5. **Observed**: Keys are present (visible via `/get_all_keys`), but some values return empty bytes.

## Expected behavior

Values offloaded to SSD should return complete original payload when fetched.

## Additional context

- Identical configuration works correctly with Mooncake code from late June, no empty‑value corruption.
- The bug emerges after upgrading to recent main‑branch commit. Could there be recent logic changes related to offload write / async persist / bucket storage backend?
- We would appreciate hints for root‑cause analysis or related commit references.

### Before submitting...

- [x] Ensure you searched for relevant issues and read the [documentation]

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the supplied master and client settings, then compare /get_all_keys results with the fetched values. Trace the bucket_storage_backend offload write and async-persist path, comparing the late-June behavior with commit 67aaf6de74f35c7c5cfd072e8643b0a80eb5999a. Done means evicted SSD-backed keys return their complete original payload rather than empty bytes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.