deepseek-ai / deepseek-ai/3FS

[RFC] IPv6 / dual-stack support (TCP control plane + RDMA data plane)

Open
#418 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.2k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

## Motivation

Some RDMA fabrics are IPv6-only: high-bandwidth NICs get ULA + SLAAC IPv6
addresses and no IPv4 at all. 3FS currently cannot run on such fabrics —
`Address` is IPv4-only by design, so neither the TCP control plane nor the
RDMA data plane can bind or dial IPv6 endpoints.

## Root cause

`src/common/utils/Address.h`:

- `uint32_t ip` + `static_assert(sizeof(Address) == sizeof(uint64_t))`;
- `operator uint64_t()` is used as the hash/equality/serialization primitive;
- `is_serde_copyable` makes RPC serialization a raw 8-byte memcpy.

Fitting a 16-byte IPv6 address requires reworking this whole chain, not a
one-line patch.

## What we have implemented and validated

We have a working dual-stack implementation (~19 files, +292/−87 against
current main), running for several weeks on a 2-node test deployment:

- `Address`: 8B → 20B (16-byte address + family), IPv6 literal parsing
(`TCP://[fd00::1]:8000`), FNV hash + explicit `operator==`, factory helpers.
- `IfAddrs`: collect `AF_INET6` addresses (skipping link-local).
- `Listener`: new `ipv6_only` option (default false, IPv4 deployments
unaffected); `ConnectionPool` / `TcpSocket` generalized via
`sockaddr_storage` with per-family branches.
- RDMA: `rdma_resolve_addr` with `sockaddr_in6`, dual-stack peer info in QP
metadata exchange, RoCE v2 IPv6 GID selection.
- Unit tests updated/added (`TestAddress` IPv6 cases, `TestEcho`,
`TestIBSocket`).

Validation: full cluster (mgmtd / meta / storage / FUSE) over IPv6 for both
TCP control plane and RDMA data plane; real LLM checkpoint loads through FUSE
at multi-GB/s per NIC; unit tests pass.

## The design question we want maintainer input on

We kept `is_serde_copyable` and simply memcpy 20 bytes. This is the smallest
change, but it makes the wire format incompatible with existing v4 binaries:
old and new nodes cannot coexist, so no rolling upgrade. Options we see:

1. Accept as a breaking change (flag day / major version bump) — simplest.
2. Move `Address` to field-based serde for cross-version compatibility —
cleaner long-term, touches RPC schema.
3. Version-negotiated dual format — most compatible, most complexity.

Which direction would you prefer? We are happy to adapt the implementation.

## Proposed contribution plan

If there is interest, we would split the work into reviewable PRs:

1. `Address` dual-stack core + unit tests;
2. TCP path (IfAddrs / Listener / ConnectionPool / TcpSocket);
3. RDMA path (IBConnect / IBSocket / GID selection).

Happy to share more benchmark details or rebase onto any preferred design.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.