aws / aws/sagemaker-python-sdk
CI: HyperPod integration tests fail due to missing hyperpod CLI and missing cluster 'riv-rig'
- 主要语言
- Python
- 星标
- 2.3k
- 派生
- 1.3k
- 平均合并
- 1 天 22 小时
- 30 天内合并 PR
- 35
描述
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
Several GPU integration tests fail in CI because the HyperPod CLI Python package/CLI is not installed and the SageMaker HyperPod cluster named "riv-rig" is not present in the test environment. Failures observed in the run https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506 include:
- ModuleNotFoundError: No module named 'hyperpod_cli'
- FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod' (the CLI executable)
- RuntimeError / botocore error: Failed to describe cluster 'riv-rig': Cluster with name riv-rig not found
Relevant failing tests:
- tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod
- tests/integ/train/test_cpt_data_mixing_hyperpod.py::test_cpt_trainer_nova_micro_with_data_mixing_hyperpod
- tests/integ/train/test_sft_data_mixing_hyperpod.py::test_sft_trainer_nova_micro_data_mixing_hyperpod
- tests/integ/train/test_nova_hyperpod.py::test_sft_trainer_nova_micro_hyperpod_lora
Stack / file references:
- Import failure originates in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py where the code tries "import hyperpod_cli" (see build_hyperpod_datamix_recipe_from_context).
- DescribeCluster failure occurs when code calls sagemaker_client.describe_cluster(ClusterName='riv-rig').
**To reproduce**
1. Check out commit (used in the failing run): 6659b11ab9440358da3b4270a2f10e2f579fb31e
2. Run the GPU integ tests (same CI job): e.g., run the workflow or locally run:
- pip install -r requirements-dev.txt
- pytest tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod -q
or run the entire integration test set used by the GPU job.
3. Observe ModuleNotFoundError or describe_cluster ResourceNotFound in the logs.
**Expected behavior**
Integration tests should not fail due to missing test dependencies or absent external infra. Possible acceptable behaviors:
- CI installs the hyperpod package/CLI before running tests, and either a test cluster exists or tests stub/skip interactions with cluster resources; or
- Tests that depend on the hyperpod CLI/cluster are mocked or skipped when running in CI without the required infra.
**Screenshots or logs**
Key excerpts:
- Module error: "ModuleNotFoundError: No module named 'hyperpod_cli'"
- Exec error: "FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod'"
- Cluster error: "RuntimeError: Failed to describe cluster 'riv-rig': An error occurred (ResourceNotFound) when calling the DescribeCluster operation: Cluster with name riv-rig not found"
Full job: https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506
**System information**
- **SageMaker Python SDK version**: repo commit 6659b11ab9440358da3b4270a2f10e2f579fb31e (CI run)
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A (integration tests for HyperPod / Nova fine-tuning)
- **Framework version**: N/A
- **Python version**: Python 3.10.13 (from CI logs)
- **CPU or GPU**: GPU integration tests (workflow is "GPU Integ Tests")
- **Custom Docker image (Y/N)**: CI uses CodeBuild / runner image (see workflow)
**Additional context**
- The test code expects the hyperpod CLI Python package (hyperpod_cli) importable and the hyperpod executable on PATH. The code that imports hyperpod_cli is in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py (build_hyperpod_datamix_recipe_from_context).
- The tests also rely on an existing HyperPod cluster named 'riv-rig' being reachable via SageMaker API. CI account does not have that cluster leading to ResourceNotFound.
- The logs also show a pip dependency warning: virtualenv requires platformdirs<4 but platformdirs 4.10.1 was installed — this is not causing the test failures but may indicate CI environment drift.
Recommended remediation
- For CI: add installation of the hyperpod package/CLI to the test setup (pip install hyperpod) or vendor a minimal fake hyperpod module for tests that only need to write recipe files.
- Make tests resilient to missing cluster:
- Use botocore.stub.Stubber or a pytest fixture to stub describe_cluster responses in CI.
- Or skip HyperPod integration tests when a real cluster isn't available (pytest.skip with a clear message).
- Prefer mocking the external infra in CI so tests are deterministic, or make infra provisioning explicit in CI docs if real cluster is required.
- Optionally pin platformdirs in test environment to avoid pip conflict warnings.
贡献指南
调研方向
从 sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py 和列出的四个测试开始,然后检查 requirements-dev.txt 和 GPU integration-test 设置。复现缺少 hyperpod_cli、hyperpod executable 和 riv-rig 时的失败。完成标准是 CI 测试能够以确定性的方式处理 CLI 和集群基础设施缺失的情况,且不会出现 ResourceNotFound 失败。
由索引模型根据 Issue 内容生成。
评估
- 技术栈
- aws, python
- 领域
- ci-cd, cloud, testing
- Issue 类型
- 缺陷
- 难度
- 4/5
- 预计耗时
- 3-5 天
- 活跃度
- 冷清
- 描述清晰度
- 基本清楚
- 新手友好度
- 48/100