lnccbrown / lnccbrown/HSSM

CI: release publish gate runs the full test suite serially in one 90-min job — reuse the 3-batch slow split

Open
#1,073 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug github_actions release
Dominant language
Python
Stars
124
Forks
24
Avg merge
19h 32m
Merged PRs (30d)
60

Description

## Problem

- The v0.4.0 publish run (29429328276) was cancelled: all three `Run full test suite / run_tests (3.12/3.13/3.14)` jobs hit `timeout-minutes: 90` inside "Run all tests". No test failed.
- `build_and_publish.yml:13-17` calls `run_tests.yml` with `run_slow_tests: true` and no `test_args`, so each Python job runs the whole suite (516 fast + ~396 slow tests) **serially**.
- Measured budget per job (per-test durations parsed from GH log timestamps of run 29419740259, py3.12, 2026-07-15): ~87.6 min slow (excl. one rerun) + ~7.8 min fast + 2–4 min setup ≈ **98–106 min > 90 min**. Deterministic timeout.
- Main stays green only because pushes run `run_slow_tests.yml`, which splits slow tests into 3 parallel batches.
- Raising the timeout is not a fix: RSS grows ~0.6 → 4.9 GiB across a single batch (native JAX/pytensor allocations; see `_slow_test_memory` in `tests/conftest.py`), so one serial full-suite job also drifts toward OOM on a 16 GB runner.

## Plan

1. Add a `workflow_call:` trigger to `run_slow_tests.yml` (keep existing `workflow_dispatch` + `push`).
2. In `build_and_publish.yml`, replace the single `run_tests` job with two:
- `run_fast_tests`: `uses: ./.github/workflows/run_tests.yml` (default fast-only path, runs `-m "not slow"`)
- `run_slow_tests`: `uses: ./.github/workflows/run_slow_tests.yml` (the existing 3-batch matrix)
3. Point the `publish` job at `needs: [lint_and_typecheck, run_fast_tests, run_slow_tests]`.
4. Verify via a pre-release tag or `workflow_dispatch` dry-run: the test gate should complete in ~35–60 min wall (longest batch today ~57 min; rebalancing is tracked separately).

## Acceptance

- A release publish completes its test gate without hitting the 90-min job timeout, using the exact batch definitions that pass on main daily.
- No change to *which* tests run — only to how they are distributed across jobs.

## Notes

- The `workflow_dispatch` skip-tests input added by #1071 (closes #1070) remains emergency-only — it publishes untested wheels.
- This is the highest-leverage, lowest-risk change of the test-speedup sequence; ship it first and independently.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading build_and_publish.yml and the existing run_tests.yml and run_slow_tests.yml workflows, especially their triggers, inputs, and test job definitions. Reuse the slow-test batch workflow for the release gate, add the fast-test gate, and update publish dependencies. Verify with a pre-release tag or workflow_dispatch dry-run that the gate finishes without the 90-minute timeout and runs the same tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
build-system, ci-cd
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.