aws / aws/sagemaker-python-sdk
CI: HyperPod integration tests fail due to missing hyperpod CLI and missing cluster 'riv-rig'
- Dominant language
- Python
- Stars
- 2.3k
- Forks
- 1.3k
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 35
Description
**PySDK Version**
- [ ] PySDK V2 (2.x)
- [x] PySDK V3 (3.x)
**Describe the bug**
Several GPU integration tests fail in CI because the HyperPod CLI Python package/CLI is not installed and the SageMaker HyperPod cluster named "riv-rig" is not present in the test environment. Failures observed in the run https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506 include:
- ModuleNotFoundError: No module named 'hyperpod_cli'
- FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod' (the CLI executable)
- RuntimeError / botocore error: Failed to describe cluster 'riv-rig': Cluster with name riv-rig not found
Relevant failing tests:
- tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod
- tests/integ/train/test_cpt_data_mixing_hyperpod.py::test_cpt_trainer_nova_micro_with_data_mixing_hyperpod
- tests/integ/train/test_sft_data_mixing_hyperpod.py::test_sft_trainer_nova_micro_data_mixing_hyperpod
- tests/integ/train/test_nova_hyperpod.py::test_sft_trainer_nova_micro_hyperpod_lora
Stack / file references:
- Import failure originates in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py where the code tries "import hyperpod_cli" (see build_hyperpod_datamix_recipe_from_context).
- DescribeCluster failure occurs when code calls sagemaker_client.describe_cluster(ClusterName='riv-rig').
**To reproduce**
1. Check out commit (used in the failing run): 6659b11ab9440358da3b4270a2f10e2f579fb31e
2. Run the GPU integ tests (same CI job): e.g., run the workflow or locally run:
- pip install -r requirements-dev.txt
- pytest tests/integ/train/test_cpt_hyperpod.py::test_cpt_trainer_nova_micro_hyperpod -q
or run the entire integration test set used by the GPU job.
3. Observe ModuleNotFoundError or describe_cluster ResourceNotFound in the logs.
**Expected behavior**
Integration tests should not fail due to missing test dependencies or absent external infra. Possible acceptable behaviors:
- CI installs the hyperpod package/CLI before running tests, and either a test cluster exists or tests stub/skip interactions with cluster resources; or
- Tests that depend on the hyperpod CLI/cluster are mocked or skipped when running in CI without the required infra.
**Screenshots or logs**
Key excerpts:
- Module error: "ModuleNotFoundError: No module named 'hyperpod_cli'"
- Exec error: "FileNotFoundError: [Errno 2] No such file or directory: 'hyperpod'"
- Cluster error: "RuntimeError: Failed to describe cluster 'riv-rig': An error occurred (ResourceNotFound) when calling the DescribeCluster operation: Cluster with name riv-rig not found"
Full job: https://github.com/aws/sagemaker-python-sdk/actions/runs/29686595481/job/88191807688#step:3:4506
**System information**
- **SageMaker Python SDK version**: repo commit 6659b11ab9440358da3b4270a2f10e2f579fb31e (CI run)
- **Framework name (eg. PyTorch) or algorithm (eg. KMeans)**: N/A (integration tests for HyperPod / Nova fine-tuning)
- **Framework version**: N/A
- **Python version**: Python 3.10.13 (from CI logs)
- **CPU or GPU**: GPU integration tests (workflow is "GPU Integ Tests")
- **Custom Docker image (Y/N)**: CI uses CodeBuild / runner image (see workflow)
**Additional context**
- The test code expects the hyperpod CLI Python package (hyperpod_cli) importable and the hyperpod executable on PATH. The code that imports hyperpod_cli is in sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py (build_hyperpod_datamix_recipe_from_context).
- The tests also rely on an existing HyperPod cluster named 'riv-rig' being reachable via SageMaker API. CI account does not have that cluster leading to ResourceNotFound.
- The logs also show a pip dependency warning: virtualenv requires platformdirs<4 but platformdirs 4.10.1 was installed — this is not causing the test failures but may indicate CI environment drift.
Recommended remediation
- For CI: add installation of the hyperpod package/CLI to the test setup (pip install hyperpod) or vendor a minimal fake hyperpod module for tests that only need to write recipe files.
- Make tests resilient to missing cluster:
- Use botocore.stub.Stubber or a pytest fixture to stub describe_cluster responses in CI.
- Or skip HyperPod integration tests when a real cluster isn't available (pytest.skip with a clear message).
- Prefer mocking the external infra in CI so tests are deterministic, or make infra provisioning explicit in CI docs if real cluster is required.
- Optionally pin platformdirs in test environment to avoid pip conflict warnings.
Contributor guide
Research direction
Start with sagemaker-train/src/sagemaker/train/common_utils/data_mixing_utils.py and the four listed tests, then inspect requirements-dev.txt and the GPU integration-test setup. Reproduce the missing hyperpod_cli, hyperpod executable, and riv-rig failures. Done means the CI tests handle missing CLI and cluster infrastructure deterministically without ResourceNotFound failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- ci-cd, cloud, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100