aws / aws/sagemaker-python-sdk

CI: HyperPod integration tests fail due to missing hyperpod CLI and missing cluster 'riv-rig'

未关闭
#6,059 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
component: training type: bug
主要语言
Python
星标
2.3k
派生
1.3k
平均合并
1 天 22 小时
30 天内合并 PR
35

描述

**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)

**Describe the bug**
Several GPU integration tests fail in CI because the HyperPod CLI Python package/CLI is not installed and the SageMaker HyperPod cluster named "riv-rig" is not present in the test environment. Failures observed in the run https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506 include:
- ModuleNotFoundError: No module named 'hyperpod_cli'
- FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod' (the CLI executable)
- RuntimeError / botocore error: Failed to describe cluster 'riv-rig': Cluster with name riv-rig not found

Relevant failing tests:
- tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod
- tests/integ/train/test_cpt_data_mixing_hyperpod.py::test_cpt_trainer_nova_micro_with_data_mixing_hyperpod
- tests/integ/train/test_sft_data_mixing_hyperpod.py::test_sft_trainer_nova_micro_data_mixing_hyperpod
- tests/integ/train/test_nova_hyperpod.py::test_sft_trainer_nova_micro_hyperpod_lora

Stack / file references:
- Import failure originates in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py where the code tries "import hyperpod_cli" (see build_hyperpod_datamix_recipe_from_context).
- DescribeCluster failure occurs when code calls sagemaker_client.describe_cluster(ClusterName='riv-rig').

**To reproduce**
1. Check out commit (used in the failing run): 6659b11ab9440358da3b4270a2f10e2f579fb31e
2. Run the GPU integ tests (same CI job): e.g., run the workflow or locally run:
- pip install -r requirements-dev.txt
- pytest tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod -q
or run the entire integration test set used by the GPU job.
3. Observe ModuleNotFoundError or describe_cluster ResourceNotFound in the logs.

**Expected behavior**
Integration tests should not fail due to missing test dependencies or absent external infra. Possible acceptable behaviors:
- CI installs the hyperpod package/CLI before running tests, and either a test cluster exists or tests stub/skip interactions with cluster resources; or
- Tests that depend on the hyperpod CLI/cluster are mocked or skipped when running in CI without the required infra.

**Screenshots or logs**
Key excerpts:
- Module error: "ModuleNotFoundError: No module named 'hyperpod_cli'"
- Exec error: "FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod'"
- Cluster error: "RuntimeError: Failed to describe cluster 'riv-rig': An error occurred (ResourceNotFound) when calling the DescribeCluster operation: Cluster with name riv-rig not found"
Full job: https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506

**System information**
- **SageMaker Python SDK version**: repo commit 6659b11ab9440358da3b4270a2f10e2f579fb31e (CI run)
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A (integration tests for HyperPod / Nova fine-tuning)
- **Framework version**: N/A
- **Python version**: Python 3.10.13 (from CI logs)
- **CPU or GPU**: GPU integration tests (workflow is "GPU Integ Tests")
- **Custom Docker image (Y/N)**: CI uses CodeBuild / runner image (see workflow)

**Additional context**
- The test code expects the hyperpod CLI Python package (hyperpod_cli) importable and the hyperpod executable on PATH. The code that imports hyperpod_cli is in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py (build_hyperpod_datamix_recipe_from_context).
- The tests also rely on an existing HyperPod cluster named 'riv-rig' being reachable via SageMaker API. CI account does not have that cluster leading to ResourceNotFound.
- The logs also show a pip dependency warning: virtualenv requires platformdirs<4 but platformdirs 4.10.1 was installed — this is not causing the test failures but may indicate CI environment drift.

Recommended remediation
- For CI: add installation of the hyperpod package/CLI to the test setup (pip install hyperpod) or vendor a minimal fake hyperpod module for tests that only need to write recipe files.
- Make tests resilient to missing cluster:
- Use botocore.stub.Stubber or a pytest fixture to stub describe_cluster responses in CI.
- Or skip HyperPod integration tests when a real cluster isn't available (pytest.skip with a clear message).
- Prefer mocking the external infra in CI so tests are deterministic, or make infra provisioning explicit in CI docs if real cluster is required.
- Optionally pin platformdirs in test environment to avoid pip conflict warnings.

贡献指南

打开贡献指南

调研方向

从 sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py 和列出的四个测试开始,然后检查 requirements-dev.txt 和 GPU integration-test 设置。复现缺少 hyperpod_cli、hyperpod executable 和 riv-rig 时的失败。完成标准是 CI 测试能够以确定性的方式处理 CLI 和集群基础设施缺失的情况,且不会出现 ResourceNotFound 失败。

由索引模型根据 Issue 内容生成。

评估

技术栈
aws, python
领域
ci-cd, cloud, testing
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
48/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。