deepseek-ai / deepseek-ai/DeepEP

ElasticBuffer asserts NCCL_GIN_TYPE_NONE on host with MLNX_OFED 24.10 (libmlx5 < MLX5_1.25)

Open
#628 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Cuda
Stars
10.1k
Forks
1.4k
Avg merge
4d 1h
Merged PRs (30d)
2

Description

DeepEP version: 2.0.0 (commit <你那 commit>)
NCCL: 2.30.4+cuda13.2
GPU: B300 SXM6, driver 590.48.01
NIC: ConnectX-7, fw_ver 40.46.5500
Host OFED: MLNX_OFED_LINUX-24.10-3.2.5.0 (libmlx5 max symbol MLX5_1.17)

NCCL_DEBUG=INFO shows:
dlvsym failed on mlx5dv_reg_dmabuf_mr ... version MLX5_1.25
dlvsym failed on mlx5dv_get_data_direct_sysfs_path ... version MLX5_1.25
Assigned GIN plugin GIN_IB_GDAKI to comm
GPU Direct RDMA Enabled for HCA 0..7
但 ncclCommQueryProperties() 返回 props.ginType = NCCL_GIN_TYPE_NONE,
ElasticBuffer ctor 在 csrc/kernels/backend/nccl.cu:87 断言失败。

Tried (all fail same):
- allow_hybrid_mode={True, False}
- NCCL_GIN_ENABLE=1
- intra-node 8-rank NVLink-only test (no inter-node fabric involved)

Question:
1. Is upgrading host MLNX_OFED to 25.x the only path? Or is there an env / NCCL flag to skip the MLX5_1.25 dmabuf
path?
2. Could README mention minimum OFED version explicitly?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.