kvcache-ai / kvcache-ai/Mooncake

[RFC]: DrainJob bandwidth rate limiting

Open
#2,875 1 comment 0 reactions 0 assignees View on GitHub
RFC
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Changes proposed

### Changes proposed

## 1. Current State: DrainJob Performance Parameters

After reading through the `mooncake-store` source, here are the existing control parameters related to DrainJob:

| Parameter | Location | Default | Purpose |
|-----------|----------|---------|---------|
| `max_concurrency` | `CreateDrainJobRequest` | 4 | Max concurrent MoveTasks per DrainJob |
| `task_thread_pool_` | Client constructor | 4 | Client-side task execution thread pool |
| `kJobDispatchThreadSleepMs` | Master dispatch loop | 500ms | Master DrainJob scheduling interval |
| TaskPollThreadMain interval | Client poll loop | 1000ms | Client task fetch interval |
| `kTaskBatchSize` | Client poll loop | 16 | Max tasks per `FetchTasks` RPC |
| `kMaxDrainUnitRetries` | Master drain logic | 3 | Max retries per key move |

**All existing controls operate on the "count" dimension. There is zero rate/bandwidth limiting.**

Key observations from the transfer path:

- In `ExecuteReplicaTransfer`, the entire object is `split_into_slices` and submitted to Transfer Engine in a single batch via `submitTransfer`. No pacing between slices.
- Drain MoveTasks share the same RDMA link and client thread pool as regular read/write operations. No priority isolation.
- A single object can be several GB. With `max_concurrency=4`, tens of GB could be in-flight on the RDMA link simultaneously.

## 2. Discussion Topic

Does the community see a need for DrainJob bandwidth/rate limiting? Are there any existing plans?

From a usage perspective, an aggressive DrainJob could saturate shared RDMA bandwidth and impact read latency for online workloads. I'd like to understand:

- Has anyone encountered this as a pain point in production?
- Any thoughts on the design direction from the community?

## 3. Possible Directions (high-level)

- **Master-side**: Track recent throughput via a sliding window (e.g., 5s) over completed task byte counts. Skip scheduling new MoveTasks when the current rate meets the configured limit. Leverages the existing 500ms dispatch loop as a natural pacing mechanism.
- **Client-side**: Split large object transfers into sub-batches (e.g., 64MB each). Insert a token bucket between sub-batches for precise pacing.
- **API**: Add an optional `bandwidth_limit_mbps` field to `CreateDrainJobRequest` (0 = unlimited, backward compatible).

The concrete design choice — single-layer vs. dual-layer, where to place it, whether to go adaptive — depends on real community needs. This RFC is intended to start that discussion.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Research direction

Start by reading the mooncake-store transfer path, especially ExecuteReplicaTransfer and submitTransfer, along with CreateDrainJobRequest and the master dispatch loop. Compare the proposed master-side and client-side approaches, then confirm the community's preferred design and API before implementation; done is not defined until that decision is made.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend-api-design, distributed-systems, networking, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.