lnccbrown / lnccbrown/HSSM

CI: test job can hang 90 min when HuggingFace download stalls during pytest collection

Open
#1,061 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
124
Forks
24
Avg merge
19h 32m
Merged PRs (30d)
60

Description

Summary

A CI test job can hang for the full timeout-minutes: 90 and end as "The operation was canceled" when the HuggingFace download of angle.onnx stalls during pytest collection, where pytest-timeout offers no protection.

Evidence (run 29286008697, job run_tests (3.14) on PR #1059)

  • All three matrix jobs started simultaneously; 3.12 and 3.13 passed in ~8.5 min. Only 3.14 hung.
  • The job log's last activity before an 89-minute silence:
21:22:04 ----------------------------- live log collection ------------------------------
21:22:05 INFO httpx: HEAD https://huggingface.co/franklab/HSSM/resolve/main/angle.onnx "HTTP/1.1 302 Found"
21:22:05 WARNING huggingface_hub: You are sending unauthenticated requests to the HF Hub...
22:51:25 ##[error]The operation was canceled.
  • Runner cleanup shows orphaned uv and pytest processes — a hang, not a crash. The workflow-level timeout-minutes: 90 is what killed it.
  • The same code (docstring-only diff) passed CI 15 minutes earlier, so this is a network flake, not a code issue.

Root cause

tests/rl/test_rl_likelihood_builder.py:33 calls make_jax_matrix_logp_funcs_from_onnx(...) at module level, so the HF download runs at import/collection time. Verified with HF_HUB_OFFLINE=1 HF_HOME=<empty> pytest --collect-only, which errors at exactly that line. pytest-timeout (360s, signal) only guards test execution, not collection — a stalled CDN connection during collection hangs the job until the 90-minute workflow timeout.

Suggested fixes

  1. Move the module-level network download in tests/rl/test_rl_likelihood_builder.py into a (module-scoped) fixture so it runs under test setup, where pytest-timeout applies — or use the local tests/fixtures/angle.onnx like tests/distribution_utils/ does.
  2. Set HF_TOKEN (repo secret) in the test workflow env — unauthenticated requests get throttled and are the likely stall trigger.
  3. Cache ~/.cache/huggingface with actions/cache keyed on the network artifact list, so matrix jobs don't re-download at all.
  4. Optionally set HF_HUB_DOWNLOAD_TIMEOUT (e.g. 30) in the workflow env so a stalled download raises quickly instead of hanging.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/rl/test_rl_likelihood_builder.py:33 and the test workflow's timeout and environment settings. Run the documented HF_HUB_OFFLINE=1 pytest --collect-only check, then inspect how tests/distribution_utils/ handles the local angle.onnx fixture. Done means pytest collection no longer hangs on the network download and the affected CI job completes within its configured timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, huggingface, python
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.