aws / aws/sagemaker-python-sdk
CI: HyperPod integration tests fail due to missing hyperpod CLI and missing cluster 'riv-rig'
- Ngôn ngữ chính
- Python
- Star
- 2.3k
- Fork
- 1.3k
- Merge trung bình
- 1 ngày 22 giờ
- Pull request đã merge (30 ngày)
- 35
Mô tả
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
Several GPU integration tests fail in CI because the HyperPod CLI Python package/CLI is not installed and the SageMaker HyperPod cluster named "riv-rig" is not present in the test environment. Failures observed in the run https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506 include:
- ModuleNotFoundError: No module named 'hyperpod_cli'
- FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod' (the CLI executable)
- RuntimeError / botocore error: Failed to describe cluster 'riv-rig': Cluster with name riv-rig not found
Relevant failing tests:
- tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod
- tests/integ/train/test_cpt_data_mixing_hyperpod.py::test_cpt_trainer_nova_micro_with_data_mixing_hyperpod
- tests/integ/train/test_sft_data_mixing_hyperpod.py::test_sft_trainer_nova_micro_data_mixing_hyperpod
- tests/integ/train/test_nova_hyperpod.py::test_sft_trainer_nova_micro_hyperpod_lora
Stack / file references:
- Import failure originates in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py where the code tries "import hyperpod_cli" (see build_hyperpod_datamix_recipe_from_context).
- DescribeCluster failure occurs when code calls sagemaker_client.describe_cluster(ClusterName='riv-rig').
**To reproduce**
1. Check out commit (used in the failing run): 6659b11ab9440358da3b4270a2f10e2f579fb31e
2. Run the GPU integ tests (same CI job): e.g., run the workflow or locally run:
- pip install -r requirements-dev.txt
- pytest tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod -q
or run the entire integration test set used by the GPU job.
3. Observe ModuleNotFoundError or describe_cluster ResourceNotFound in the logs.
**Expected behavior**
Integration tests should not fail due to missing test dependencies or absent external infra. Possible acceptable behaviors:
- CI installs the hyperpod package/CLI before running tests, and either a test cluster exists or tests stub/skip interactions with cluster resources; or
- Tests that depend on the hyperpod CLI/cluster are mocked or skipped when running in CI without the required infra.
**Screenshots or logs**
Key excerpts:
- Module error: "ModuleNotFoundError: No module named 'hyperpod_cli'"
- Exec error: "FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod'"
- Cluster error: "RuntimeError: Failed to describe cluster 'riv-rig': An error occurred (ResourceNotFound) when calling the DescribeCluster operation: Cluster with name riv-rig not found"
Full job: https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506
**System information**
- **SageMaker Python SDK version**: repo commit 6659b11ab9440358da3b4270a2f10e2f579fb31e (CI run)
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A (integration tests for HyperPod / Nova fine-tuning)
- **Framework version**: N/A
- **Python version**: Python 3.10.13 (from CI logs)
- **CPU or GPU**: GPU integration tests (workflow is "GPU Integ Tests")
- **Custom Docker image (Y/N)**: CI uses CodeBuild / runner image (see workflow)
**Additional context**
- The test code expects the hyperpod CLI Python package (hyperpod_cli) importable and the hyperpod executable on PATH. The code that imports hyperpod_cli is in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py (build_hyperpod_datamix_recipe_from_context).
- The tests also rely on an existing HyperPod cluster named 'riv-rig' being reachable via SageMaker API. CI account does not have that cluster leading to ResourceNotFound.
- The logs also show a pip dependency warning: virtualenv requires platformdirs<4 but platformdirs 4.10.1 was installed — this is not causing the test failures but may indicate CI environment drift.
Recommended remediation
- For CI: add installation of the hyperpod package/CLI to the test setup (pip install hyperpod) or vendor a minimal fake hyperpod module for tests that only need to write recipe files.
- Make tests resilient to missing cluster:
- Use botocore.stub.Stubber or a pytest fixture to stub describe_cluster responses in CI.
- Or skip HyperPod integration tests when a real cluster isn't available (pytest.skip with a clear message).
- Prefer mocking the external infra in CI so tests are deterministic, or make infra provisioning explicit in CI docs if real cluster is required.
- Optionally pin platformdirs in test environment to avoid pip conflict warnings.
Hướng dẫn đóng góp
Hướng nghiên cứu
Bắt đầu với sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py và bốn bài kiểm thử được liệt kê, sau đó kiểm tra requirements-dev.txt và thiết lập GPU integration-test. Tái hiện các lỗi khi thiếu hyperpod_cli, hyperpod executable và riv-rig. Hoàn tất khi các bài kiểm thử CI xử lý một cách xác định việc thiếu CLI và hạ tầng cluster mà không phát sinh lỗi ResourceNotFound.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- aws, python
- Lĩnh vực
- ci-cd, cloud, testing
- Loại issue
- Lỗi
- Độ khó
- 4/5
- Thời gian dự kiến
- 3-5 ngày
- Mức độ hoạt động
- Ít trao đổi
- Độ rõ ràng
- Khá rõ ràng
- Mức phù hợp với người mới
- 48/100