aws / aws/sagemaker-python-sdk

Add ml.p5e.48xlarge to EFA instance lists in sagemaker-train and sagemaker-core

未关闭
#5,491 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

# Add ml.p5e.48xlarge to EFA instance lists in sagemaker-train and sagemaker-core

## Description

The `SM_EFA_NCCL_INSTANCES` and `SM_EFA_RDMA_INSTANCES` lists in the sagemaker-python-sdk are missing `ml.p5e.48xlarge`, causing NCCL hangs during distributed training initialization on P5e instances when using the SDK's container drivers.

Additionally, `ml.p5.48xlarge` is missing from `SM_EFA_RDMA_INSTANCES` (it's only in `SM_EFA_NCCL_INSTANCES`).

## Current State

```python
SM_EFA_NCCL_INSTANCES = [
"ml.g4dn.8xlarge",
"ml.g4dn.12xlarge",
"ml.g5.48xlarge",
"ml.p3dn.24xlarge",
"ml.p4d.24xlarge",
"ml.p4de.24xlarge",
"ml.p5.48xlarge",
"ml.trn1.32xlarge",
]

SM_EFA_RDMA_INSTANCES = [
"ml.p4d.24xlarge",
"ml.p4de.24xlarge",
"ml.trn1.32xlarge",
]
```

## Expected State

```python
SM_EFA_NCCL_INSTANCES = [
"ml.g4dn.8xlarge",
"ml.g4dn.12xlarge",
"ml.g5.48xlarge",
"ml.p3dn.24xlarge",
"ml.p4d.24xlarge",
"ml.p4de.24xlarge",
"ml.p5.48xlarge",
"ml.p5e.48xlarge", # ADD
"ml.trn1.32xlarge",
]

SM_EFA_RDMA_INSTANCES = [
"ml.p4d.24xlarge",
"ml.p4de.24xlarge",
"ml.p5.48xlarge", # ADD
"ml.p5e.48xlarge", # ADD
"ml.trn1.32xlarge",
]
```

## Impact

Without these entries, the SDK's container drivers don't set the required EFA environment variables (`FI_PROVIDER=efa`, `FI_EFA_USE_DEVICE_RDMA=1`, `RDMAV_FORK_SAFE=1`) for P5e instances, causing NCCL to hang during collective initialization in multi-node distributed training.

## Related

- sagemaker-training-toolkit issue: https://github.com/aws/sagemaker-training-toolkit/issues/240
- sagemaker-training-toolkit PR: https://github.com/aws/sagemaker-training-toolkit/pull/241
- P5e instances use EFA with RDMA support, same as P4d/P4de/P5

## Questions

1. Is there a specific process for testing EFA/instance-specific changes on actual hardware before merging?
2. Should integration tests be added for P5e EFA configuration, or are unit tests sufficient?

贡献指南

打开贡献指南

调研方向

首先定位 sagemaker-train 和 sagemaker-core 组件中的 SM_EFA_NCCL_INSTANCES 和 SM_EFA_RDMA_INSTANCES 定义。将两个列表与此 issue 中的预期状态进行比较,然后检查相关的单元测试覆盖率;完成的标准是两个 P5 实例都出现在相应的 EFA 列表中。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
cloud, machine-learning
Issue 类型
缺陷
难度
2/5
预计耗时
1-3 小时
活跃度
停滞
描述清晰度
描述清楚
新手友好度
58/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。