kvcache-ai / kvcache-ai/Mooncake
[Feature Request]: Support multiple DFS roots for NFS shard placement
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
### Describe your feature request
## Motivation
This feature primarily targets deployments where NFS multipath or equivalent
cross-endpoint traffic striping is unavailable.
A storage server may expose multiple NFS endpoints through different IP
addresses and NICs. Compute nodes can mount each endpoint separately, but
Mooncake currently supports only one `MOONCAKE_DFS_ROOT_DIR`, so all DFS shards
and traffic use a single mount.
For example:
```text
10.0.0.10:/export/0 -> /mnt/mooncake/nfs0
10.0.1.10:/export/1 -> /mnt/mooncake/nfs1
```
## Proposed change
Allow descriptor-based DFS storage to configure multiple mounted roots:
```bash
MOONCAKE_DFS_ROOT_DIRS=/mnt/mooncake/nfs0,/mnt/mooncake/nfs1
```
Distribute shard files deterministically across them, for example:
```text
root_index = shard_index % root_count
```
The master and all clients must use the same ordered root list and mount paths.
Mooncake only manages shard placement; mounting NFS exports and selecting their
IP/NIC paths remain deployment responsibilities.
Keep `MOONCAKE_DFS_ROOT_DIR` as the backward-compatible single-root fallback.
## Constraints
- Support multiple roots only with the POSIX filesystem adapter, primarily for
pre-mounted NFS exports. HF3FS multi-root support is out of scope.
- Require every configured root to be an existing, absolute, and unique path.
Initialization must fail if a root is unavailable.
- Treat the ordered root list as immutable storage layout metadata. The master
and every client must use identical root counts, ordering, and local mount
paths. Changing the list after shard files have been created is unsupported.
- Use equal-weight round-robin placement only. Roots are expected to provide
comparable usable capacity.
### Before submitting a new issue...
- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)
Contributor guide
Research direction
Start by locating the MOONCAKE_DFS_ROOT_DIR configuration and the POSIX filesystem adapter, then trace how the master and clients choose DFS shard paths. Done means validating an ordered, absolute, unique root list, preserving the single-root fallback, and placing shards deterministically across roots while leaving HF3FS unchanged.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100