NVIDIA-NeMo / NVIDIA-NeMo/Curator
ci: runtime_env tests exhaust /tmp on CPU runners due to Ray venv cloning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.8k
- Forks
- 328
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 30
Description
Problem
Tests that use per-stage runtime_env (pip/uv package overrides) cannot run on GitHub-hosted CPU runners because they exhaust /tmp disk space.
Root cause: For each unique runtime_env spec, Ray clones the entire .venv into /tmp via shutil.copytree (see ray/_private/runtime_env/_clonevirtualenv.py). Because the venv includes NVIDIA CUDA libraries (libcudnn_engines_precompiled.so.9, libnccl.so.2, etc.), each clone is ~700 MB+. With 3 unique specs the test suite needs ~2.1 GB in /tmp, which exhausts available space on CPU runners.
Observed error:
```
shutil.Error: [Errno 28] No space left on device:
libcudnn_engines_precompiled.so.9 →
/tmp/pytest-of-runner/pytest-0/ray0/session_.../runtime_resources/uv/.../virtualenv/...
```
Workaround: The affected tests (tests/pipelines/test_per_stage_runtime_env.py, tests/pipelines/test_runtime_env_advanced.py) are currently marked @pytest.mark.gpu so they run only on GPU runners, which have more available disk. See PR #1623.
Fix options
-
Point Ray's temp dir to the workspace — pass
--basetemp=$GITHUB_WORKSPACE/pytest-tmpto pytest sotmp_path_factory.mktemp("ray")(used asray start --temp-dir) lands on the larger/home/runner/workpartition instead of/tmp. -
Exclude NVIDIA libs from the venv clone — Ray's
_clonevirtualenv.pyusesshutil.copytree(..., ignore=shutil.ignore_patterns("*.pyc")). Patching this (or thevirtualenv_utils.pycaller) to also ignorenvidia/packages would eliminate the bulk of the clone size. -
Use a CPU-only venv for CI — exclude
nvidia-*packages from the install. Bigger change but permanently avoids the issue.
Option 1 is the least invasive and can be done entirely in .github/workflows/cicd-main.yml.
Acceptance criteria
tests/pipelines/test_per_stage_runtime_env.pyandtests/pipelines/test_runtime_env_advanced.pypass in CPU CI@pytest.mark.gpumarkers removed from those files
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in .github/workflows/cicd-main.yml and inspect how pytest is invoked for CPU runners, then review the markers in tests/pipelines/test_per_stage_runtime_env.py and tests/pipelines/test_runtime_env_advanced.py. Run those two test files in the CPU workflow after applying the workflow change; done means both pass there and their @pytest.mark.gpu markers are removed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 74/100