kvcache-ai / kvcache-ai/Mooncake
[RFC]: DrainJob bandwidth rate limiting
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Changes proposed
### Changes proposed
## 1. Current State: DrainJob Performance Parameters
After reading through the `mooncake-store` source, here are the existing control parameters related to DrainJob:
| Parameter | Location | Default | Purpose |
|-----------|----------|---------|---------|
| `max_concurrency` | `CreateDrainJobRequest` | 4 | Max concurrent MoveTasks per DrainJob |
| `task_thread_pool_` | Client constructor | 4 | Client-side task execution thread pool |
| `kJobDispatchThreadSleepMs` | Master dispatch loop | 500ms | Master DrainJob scheduling interval |
| TaskPollThreadMain interval | Client poll loop | 1000ms | Client task fetch interval |
| `kTaskBatchSize` | Client poll loop | 16 | Max tasks per `FetchTasks` RPC |
| `kMaxDrainUnitRetries` | Master drain logic | 3 | Max retries per key move |
**All existing controls operate on the "count" dimension. There is zero rate/bandwidth limiting.**
Key observations from the transfer path:
- In `ExecuteReplicaTransfer`, the entire object is `split_into_slices` and submitted to Transfer Engine in a single batch via `submitTransfer`. No pacing between slices.
- Drain MoveTasks share the same RDMA link and client thread pool as regular read/write operations. No priority isolation.
- A single object can be several GB. With `max_concurrency=4`, tens of GB could be in-flight on the RDMA link simultaneously.
## 2. Discussion Topic
Does the community see a need for DrainJob bandwidth/rate limiting? Are there any existing plans?
From a usage perspective, an aggressive DrainJob could saturate shared RDMA bandwidth and impact read latency for online workloads. I'd like to understand:
- Has anyone encountered this as a pain point in production?
- Any thoughts on the design direction from the community?
## 3. Possible Directions (high-level)
- **Master-side**: Track recent throughput via a sliding window (e.g., 5s) over completed task byte counts. Skip scheduling new MoveTasks when the current rate meets the configured limit. Leverages the existing 500ms dispatch loop as a natural pacing mechanism.
- **Client-side**: Split large object transfers into sub-batches (e.g., 64MB each). Insert a token bucket between sub-batches for precise pacing.
- **API**: Add an optional `bandwidth_limit_mbps` field to `CreateDrainJobRequest` (0 = unlimited, backward compatible).
The concrete design choice — single-layer vs. dual-layer, where to place it, whether to go adaptive — depends on real community needs. This RFC is intended to start that discussion.
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Research direction
Start by reading the mooncake-store transfer path, especially ExecuteReplicaTransfer and submitTransfer, along with CreateDrainJobRequest and the master dispatch loop. Compare the proposed master-side and client-side approaches, then confirm the community's preferred design and API before implementation; done is not defined until that decision is made.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend-api-design, distributed-systems, networking, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100