kvcache-ai / kvcache-ai/Mooncake
[Bug]: Silent offload failure: keys exist on offload SSD but return empty value after eviction offload
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Bug Report
### Describe the bug
We observe silent offload failure against local SSD offload backend. Keys are visible via `/get_all_keys`, but corresponding values return empty when fetched.
We are testing with recent Mooncake commit: https://github.com/kvcache-ai/Mooncake/commit/67aaf6de74f35c7c5cfd072e8643b0a80eb5999a (2026‑08‑12).
The same workload with June‑end Mooncake code works perfectly without empty‑value issue. This regression appears after we upgraded to newer codebase.
### Master launch command:
```C++
mooncake_master \
--rpc_port=8001 \
--rpc_thread_num=8 \
--metrics_port=8007 \
--default_kv_lease_ttl=5000 \
--allow_evict_soft_pinned_objects=true \
--eviction_ratio=0.35 \
--eviction_high_watermark_ratio=0.7 \
--cluster_id=mooncake_cluster \
--allocation_strategy=free_ratio_first \
--quota_bytes=3298534883328 \
--enable_ha=false \
--enable_offload=true \
--offload_on_evict=true \
--offload_force_evict=false \
--enable_disk_eviction=true \
--promotion_on_hit=false \
--put_start_discard_timeout_sec=300 \
--put_start_release_timeout_sec=600
```
### Real‑Client Environment Variables & Settings
``` C++
MC_IB_PCI_RELAXED_ORDERING = 1
MOONCAKE_LOCAL_BUFFER_SIZE = 6442450944
MOONCAKE_OFFLOAD_BUCKET_EVICTION_POLICY = lru
MOONCAKE_OFFLOAD_BUCKET_KEYS_LIMIT = 128
MOONCAKE_OFFLOAD_BUCKET_MAX_TOTAL_SIZE = 3113851289600
MOONCAKE_OFFLOAD_BUCKET_SIZE_LIMIT_BYTES = 268435456
MOONCAKE_OFFLOAD_CLIENT_BUFFER_GC_INTERVAL_SECONDS = 2
MOONCAKE_OFFLOAD_FILE_STORAGE_PATH = /ssd_file/mooncake_cache/
MOONCAKE_OFFLOAD_HEARTBEAT_INTERVAL_SECONDS = 1
MOONCAKE_OFFLOAD_LOCAL_BUFFER_SIZE_BYTES = 21474836480
MOONCAKE_OFFLOAD_STORAGE_BACKEND_DESCRIPTOR = bucket_storage_backend
MOONCAKE_OFFLOAD_TOTAL_SIZE_LIMIT_BYTES = 3221225472000
MOONCAKE_OFFLOAD_USE_URING = 1
```
Client launch command:
```C++
mooncake_client \
--master_server_address=:8001 \
--host= \
--protocol=rdma \
--port=8005 \
--global_segment_size=25GB \
--enable_offload=true \
--threads=8 \
--metadata_server=P2PHANDSHAKE
```
## To Reproduce
1. Start master and real‑client with above configurations, `offload_on_evict=true`.
2. Write ~100GB total data. Logs confirm all keys‑values are successfully written into memory.
3. With `offload_on_evict=true`, objects should be offloaded to local SSD and persist. No heavy SSD‑side eviction is expected under this workload.
4. After writing completes, read back keys.
5. **Observed**: Keys are present (visible via `/get_all_keys`), but some values return empty bytes.
## Expected behavior
Values offloaded to SSD should return complete original payload when fetched.
## Additional context
- Identical configuration works correctly with Mooncake code from late June, no empty‑value corruption.
- The bug emerges after upgrading to recent main‑branch commit. Could there be recent logic changes related to offload write / async persist / bucket storage backend?
- We would appreciate hints for root‑cause analysis or related commit references.
### Before submitting...
- [x] Ensure you searched for relevant issues and read the [documentation]
Contributor guide
Research direction
Reproduce the issue with the supplied master and client settings, then compare /get_all_keys results with the fetched values. Trace the bucket_storage_backend offload write and async-persist path, comparing the late-June behavior with commit 67aaf6de74f35c7c5cfd072e8643b0a80eb5999a. Done means evicted SSD-backed keys return their complete original payload rather than empty bytes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100