CI: test job can hang 90 min when HuggingFace download stalls during pytest collection
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 24
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 60
Description
Summary
A CI test job can hang for the full timeout-minutes: 90 and end as "The operation was canceled" when the HuggingFace download of angle.onnx stalls during pytest collection, where pytest-timeout offers no protection.
Evidence (run 29286008697, job run_tests (3.14) on PR #1059)
- All three matrix jobs started simultaneously; 3.12 and 3.13 passed in ~8.5 min. Only 3.14 hung.
- The job log's last activity before an 89-minute silence:
21:22:04 ----------------------------- live log collection ------------------------------
21:22:05 INFO httpx: HEAD https://huggingface.co/franklab/HSSM/resolve/main/angle.onnx "HTTP/1.1 302 Found"
21:22:05 WARNING huggingface_hub: You are sending unauthenticated requests to the HF Hub...
22:51:25 ##[error]The operation was canceled.
- Runner cleanup shows orphaned
uvandpytestprocesses — a hang, not a crash. The workflow-leveltimeout-minutes: 90is what killed it. - The same code (docstring-only diff) passed CI 15 minutes earlier, so this is a network flake, not a code issue.
Root cause
tests/rl/test_rl_likelihood_builder.py:33 calls make_jax_matrix_logp_funcs_from_onnx(...) at module level, so the HF download runs at import/collection time. Verified with HF_HUB_OFFLINE=1 HF_HOME=<empty> pytest --collect-only, which errors at exactly that line. pytest-timeout (360s, signal) only guards test execution, not collection — a stalled CDN connection during collection hangs the job until the 90-minute workflow timeout.
Suggested fixes
- Move the module-level network download in
tests/rl/test_rl_likelihood_builder.pyinto a (module-scoped) fixture so it runs under test setup, wherepytest-timeoutapplies — or use the localtests/fixtures/angle.onnxliketests/distribution_utils/does. - Set
HF_TOKEN(repo secret) in the test workflow env — unauthenticated requests get throttled and are the likely stall trigger. - Cache
~/.cache/huggingfacewithactions/cachekeyed on the network artifact list, so matrix jobs don't re-download at all. - Optionally set
HF_HUB_DOWNLOAD_TIMEOUT(e.g. 30) in the workflow env so a stalled download raises quickly instead of hanging.
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/rl/test_rl_likelihood_builder.py:33 and the test workflow's timeout and environment settings. Run the documented HF_HUB_OFFLINE=1 pytest --collect-only check, then inspect how tests/distribution_utils/ handles the local angle.onnx fixture. Done means pytest collection no longer hangs on the network download and the affected CI job completes within its configured timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, huggingface, python
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100