aws / aws/sagemaker-python-sdk

Add ml.p5e.48xlarge to EFA instance lists in sagemaker-train and sagemaker-core

Offen
#5,491 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
2.3k
Forks
1.3k
Ø Merge
1 T. 22 Std.
Gemergte PRs (30 T.)
35

Beschreibung

# Add ml.p5e.48xlarge to EFA instance lists in sagemaker-train and sagemaker-core

## Description

The `SM_EFA_NCCL_INSTANCES` and `SM_EFA_RDMA_INSTANCES` lists in the sagemaker-python-sdk are missing `ml.p5e.48xlarge`, causing NCCL hangs during distributed training initialization on P5e instances when using the SDK's container drivers.

Additionally, `ml.p5.48xlarge` is missing from `SM_EFA_RDMA_INSTANCES` (it's only in `SM_EFA_NCCL_INSTANCES`).

## Current State

```python
SM_EFA_NCCL_INSTANCES = [
"ml.g4dn.8xlarge",
"ml.g4dn.12xlarge",
"ml.g5.48xlarge",
"ml.p3dn.24xlarge",
"ml.p4d.24xlarge",
"ml.p4de.24xlarge",
"ml.p5.48xlarge",
"ml.trn1.32xlarge",
]

SM_EFA_RDMA_INSTANCES = [
"ml.p4d.24xlarge",
"ml.p4de.24xlarge",
"ml.trn1.32xlarge",
]
```

## Expected State

```python
SM_EFA_NCCL_INSTANCES = [
"ml.g4dn.8xlarge",
"ml.g4dn.12xlarge",
"ml.g5.48xlarge",
"ml.p3dn.24xlarge",
"ml.p4d.24xlarge",
"ml.p4de.24xlarge",
"ml.p5.48xlarge",
"ml.p5e.48xlarge", # ADD
"ml.trn1.32xlarge",
]

SM_EFA_RDMA_INSTANCES = [
"ml.p4d.24xlarge",
"ml.p4de.24xlarge",
"ml.p5.48xlarge", # ADD
"ml.p5e.48xlarge", # ADD
"ml.trn1.32xlarge",
]
```

## Impact

Without these entries, the SDK's container drivers don't set the required EFA environment variables (`FI_PROVIDER=efa`, `FI_EFA_USE_DEVICE_RDMA=1`, `RDMAV_FORK_SAFE=1`) for P5e instances, causing NCCL to hang during collective initialization in multi-node distributed training.

## Related

- sagemaker-training-toolkit issue: https://github.com/aws/sagemaker-training-toolkit/issues/240
- sagemaker-training-toolkit PR: https://github.com/aws/sagemaker-training-toolkit/pull/241
- P5e instances use EFA with RDMA support, same as P4d/P4de/P5

## Questions

1. Is there a specific process for testing EFA/instance-specific changes on actual hardware before merging?
2. Should integration tests be added for P5e EFA configuration, or are unit tests sufficient?

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Beginne damit, die Definitionen von SM_EFA_NCCL_INSTANCES und SM_EFA_RDMA_INSTANCES in den Komponenten sagemaker-train und sagemaker-core zu finden. Vergleiche beide Listen mit dem in diesem Issue erwarteten Zustand und überprüfe anschließend die relevante Unit-Test-Abdeckung; abgeschlossen bedeutet, dass beide P5-Instanzen in den jeweils passenden EFA-Listen enthalten sind.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
aws, python
Bereich
cloud, machine-learning
Issue-Typ
Bug
Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Aktivitätsstatus
Veraltet
Klarheit
Klar beschrieben
Anfängerfreundlichkeit
58/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.