antirez / antirez/ds4

Distributed pipeline crash / layer routing limitation: coordinator crashes without output layer, but --layers cannot express non-contiguous spans

Open
#610 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
22.3k
Forks
2.1k
Avg merge
1d 3h
Merged PRs (30d)
4

Description

Environment

ds4 commit: efdadd41e20134af4f3381e1ed90e96fe4faef6f (and also tested after fetch showing newer 0a7ad77, please confirm if fix is expected there)
Platform: Linux
GPU: NVIDIA GB10 (sm_121)
2 nodes (DGX1 + DGX2), testing distributed pipeline
Backend: CUDA
What I’m trying to do Split model across two machines:

coordinator on DGX1 for early layers
worker on DGX2 for later layers
Example intent:

coordinator: 0:20
worker: 21:output
Observed behavior

Coordinator with early-layer span crashes at startup:
bash
/home/administrator/ds4/bin/ds4-server \
--role coordinator \
--layers 0:20 \
--listen 0.0.0.0 9000 \
--host 0.0.0.0 \
--port 8000 \
--ctx 8192
Logs end with:

model map restricted to 0:20
memory planning completes
Segmentation fault (core dumped)
If trying to include output with non-contiguous syntax, parser rejects it:
bash
--layers 0:20,output
Error:

Text
invalid --layers 0:20,output: invalid end layer in 0:20,output
Worker rejects coordinator-only prefill flags (expected, just noting):
Text
--dist-prefill-chunk requires --role coordinator
When testing API during broken routing, sometimes get:
JSON
{"error":{"message":"coordinator route does not start at layer 0","type":"invalid_request_error"}}
Why this seems problematic If coordinator runtime effectively requires owning output/final stage in this configuration, but --layers only accepts contiguous spans and cannot express 0:K + output, then a valid split topology may be impossible to express via CLI.

Expected behavior At least one of:

Coordinator should work with contiguous 0:K while worker handles K+1:output (no segfault), or
CLI should support disjoint layer assignment (e.g. 0:20,output) if coordinator needs output stage, or
A clear documented constraint and recommended valid topology for multi-node split on CUDA/GB10.
Request

Please confirm whether this is a bug or intended limitation.
If bug: any patch/workaround?
If limitation: can disjoint --layers support be added, or can coordinator ownership rules be relaxed?
Additional notes

Standalone single-node tensor-parallel mode works on both nodes.
Problem appears specific to distributed coordinator path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.