kvcache-ai / kvcache-ai/Mooncake

[Feature Request]: Support multiple DFS roots for NFS shard placement

Open
#3,815 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

### Describe your feature request

## Motivation

This feature primarily targets deployments where NFS multipath or equivalent
cross-endpoint traffic striping is unavailable.

A storage server may expose multiple NFS endpoints through different IP
addresses and NICs. Compute nodes can mount each endpoint separately, but
Mooncake currently supports only one `MOONCAKE_DFS_ROOT_DIR`, so all DFS shards
and traffic use a single mount.

For example:

```text
10.0.0.10:/export/0 -> /mnt/mooncake/nfs0
10.0.1.10:/export/1 -> /mnt/mooncake/nfs1
```

## Proposed change

Allow descriptor-based DFS storage to configure multiple mounted roots:

```bash
MOONCAKE_DFS_ROOT_DIRS=/mnt/mooncake/nfs0,/mnt/mooncake/nfs1
```

Distribute shard files deterministically across them, for example:

```text
root_index = shard_index % root_count
```

The master and all clients must use the same ordered root list and mount paths.
Mooncake only manages shard placement; mounting NFS exports and selecting their
IP/NIC paths remain deployment responsibilities.

Keep `MOONCAKE_DFS_ROOT_DIR` as the backward-compatible single-root fallback.

## Constraints

- Support multiple roots only with the POSIX filesystem adapter, primarily for
pre-mounted NFS exports. HF3FS multi-root support is out of scope.
- Require every configured root to be an existing, absolute, and unique path.
Initialization must fail if a root is unavailable.
- Treat the ordered root list as immutable storage layout metadata. The master
and every client must use identical root counts, ordering, and local mount
paths. Changing the list after shard files have been created is unsupported.
- Use equal-weight round-robin placement only. Roots are expected to provide
comparable usable capacity.

### Before submitting a new issue...

- [x] Make sure you already searched for relevant issues and read the [documentation](https://kvcache-ai.github.io/Mooncake/)

Contributor guide

Open the contributing guide

Research direction

Start by locating the MOONCAKE_DFS_ROOT_DIR configuration and the POSIX filesystem adapter, then trace how the master and clients choose DFS shard paths. Done means validating an ordered, absolute, unique root list, preserving the single-root fallback, and placing shards deterministically across roots while leaving HF3FS unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.