deepseek-ai / deepseek-ai/DeepEP

When running test_internode.py on two nodes (with 8 GPUs per node), an error occurs.

Open
#503 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

When running test_internode.py on two nodes (with 8 GPUs per node), an error occurs.
Execution Command::

> MASTER_ADDR=${MASTER_ADDR:-"10.124.173.185"}
> MASTER_PORT=${MASTER_PORT:-10556}
> WORLD_SIZE=${WORLD_SIZE:-2}
> RANK=${RANK:-${ARNOLD_ID:-0}}
>
> MASTER_ADDR="$MASTER_ADDR" MASTER_PORT=$MASTER_PORT WORLD_SIZE=$WORLD_SIZE RANK=$RANK \
> python ./tests/test_internode.py 2>&1 | tee debug_${RANK}.log

Error Logs:
rank0:

https://github.com/SaltFish11/DeepEP/blob/develop/debug_0.log

rank1:

https://github.com/SaltFish11/DeepEP/blob/develop/debug_1.log

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.