aws / aws/sagemaker-python-sdk

CI: HyperPod integration tests fail due to missing hyperpod CLI and missing cluster 'riv-rig'

Aperta
#6,059 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
component: training type: bug
Lingua principale
Python
Stelle
2.3k
Fork
1.3k
Merge medio
1g 22h
PR unite (30g)
35

Descrizione

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

**Describe the bug**
Several GPU integration tests fail in CI because the HyperPod CLI Python package/CLI is not installed and the SageMaker HyperPod cluster named "riv-rig" is not present in the test environment. Failures observed in the run https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506 include:
- ModuleNotFoundError: No module named 'hyperpod_cli'
- FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod' (the CLI executable)
- RuntimeError / botocore error: Failed to describe cluster 'riv-rig': Cluster with name riv-rig not found

Relevant failing tests:
- tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod
- tests/integ/train/test_cpt_data_mixing_hyperpod.py::test_cpt_trainer_nova_micro_with_data_mixing_hyperpod
- tests/integ/train/test_sft_data_mixing_hyperpod.py::test_sft_trainer_nova_micro_data_mixing_hyperpod
- tests/integ/train/test_nova_hyperpod.py::test_sft_trainer_nova_micro_hyperpod_lora

Stack / file references:
- Import failure originates in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py where the code tries "import hyperpod_cli" (see build_hyperpod_datamix_recipe_from_context).
- DescribeCluster failure occurs when code calls sagemaker_client.describe_cluster(ClusterName='riv-rig').

**To reproduce**
1. Check out commit (used in the failing run): 6659b11ab9440358da3b4270a2f10e2f579fb31e
2. Run the GPU integ tests (same CI job): e.g., run the workflow or locally run:
- pip install -r requirements-dev.txt
- pytest tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod -q
or run the entire integration test set used by the GPU job.
3. Observe ModuleNotFoundError or describe_cluster ResourceNotFound in the logs.

**Expected behavior**
Integration tests should not fail due to missing test dependencies or absent external infra. Possible acceptable behaviors:
- CI installs the hyperpod package/CLI before running tests, and either a test cluster exists or tests stub/skip interactions with cluster resources; or
- Tests that depend on the hyperpod CLI/cluster are mocked or skipped when running in CI without the required infra.

**Screenshots or logs**
Key excerpts:
- Module error: "ModuleNotFoundError: No module named 'hyperpod_cli'"
- Exec error: "FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod'"
- Cluster error: "RuntimeError: Failed to describe cluster 'riv-rig': An error occurred (ResourceNotFound) when calling the DescribeCluster operation: Cluster with name riv-rig not found"
Full job: https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506

**System information**
- **SageMaker Python SDK version**: repo commit 6659b11ab9440358da3b4270a2f10e2f579fb31e (CI run)
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A (integration tests for HyperPod / Nova fine-tuning)
- **Framework version**: N/A
- **Python version**: Python 3.10.13 (from CI logs)
- **CPU or GPU**: GPU integration tests (workflow is "GPU Integ Tests")
- **Custom Docker image (Y/N)**: CI uses CodeBuild / runner image (see workflow)

**Additional context**
- The test code expects the hyperpod CLI Python package (hyperpod_cli) importable and the hyperpod executable on PATH. The code that imports hyperpod_cli is in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py (build_hyperpod_datamix_recipe_from_context).
- The tests also rely on an existing HyperPod cluster named 'riv-rig' being reachable via SageMaker API. CI account does not have that cluster leading to ResourceNotFound.
- The logs also show a pip dependency warning: virtualenv requires platformdirs<4 but platformdirs 4.10.1 was installed — this is not causing the test failures but may indicate CI environment drift.

Recommended remediation
- For CI: add installation of the hyperpod package/CLI to the test setup (pip install hyperpod) or vendor a minimal fake hyperpod module for tests that only need to write recipe files.
- Make tests resilient to missing cluster:
- Use botocore.stub.Stubber or a pytest fixture to stub describe_cluster responses in CI.
- Or skip HyperPod integration tests when a real cluster isn't available (pytest.skip with a clear message).
- Prefer mocking the external infra in CI so tests are deterministic, or make infra provisioning explicit in CI docs if real cluster is required.
- Optionally pin platformdirs in test environment to avoid pip conflict warnings.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia da sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py e dai quattro test elencati, quindi esamina requirements-dev.txt e la configurazione di GPU integration-test. Riproduci i fallimenti dovuti alla mancanza di hyperpod_cli, hyperpod executable e riv-rig. Il lavoro è completato quando i test CI gestiscono in modo deterministico l’assenza della CLI e dell’infrastruttura del cluster, senza errori ResourceNotFound.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, python
Ambito
ci-cd, cloud, testing
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.