lnccbrown / lnccbrown/HSSM

Tracking: eliminate the release-CI timeout and speed up the test suite

Open
#1,083 2 comments 0 reactions 0 assignees View on GitHub
github_actions release
Dominant language
Python
Stars
124
Forks
24
Avg merge
19h 32m
Merged PRs (30d)
60

Description

## Context

The v0.4.0 publish (run 29429328276, 2026-07-15) was cancelled because the release workflow runs the **entire suite serially in one 90-min job** per Python version, while slow tests alone sum to ~96 min (measured from run 29419740259 logs, py3.12). Main only looks green because pushes use the 3-batch split in `run_slow_tests.yml`. Full analysis: a 7-agent audit of `tests/`, `.github/workflows/`, and `pyproject.toml`, cross-checked against measured per-test CI durations.

Key facts driving the plan:
- One test (`test_addm_waic_loo`) deterministically exceeds `--timeout=360` and reruns untimed → 16.6 min every run.
- The slow suite is a sampler-dispatch smoke matrix (a 20-row grid cloned across 3 files; only ~9 rows exercise distinct HSSM code paths); statistical correctness lives in fast numeric tests.
- `coverage.yml` redundantly re-runs the full suite (~2 h) on every main push; `--cov=.` instruments every job whose report is never uploaded.
- Batches are unbalanced 19.8/22.9/**53.3** min; RSS grows to ~4.9 GiB per batch (so raising the timeout is not an option).

## Sequence

**Phase 0 — unblock releases (config-only, ship first)**
- [ ] #1073 Batch the release test gate (reuse `run_slow_tests.yml` via `workflow_call`)

**Phase 1 — config & CI hygiene (independent, ~1 day total)**
- [ ] #1074 waic_loo timeout pathology (Step 1: timeout marker; Step 2: synthetic-posterior rewrite)
- [ ] #1075 pytest addopts cleanup (+ fix stale `--runslow` docs)
- [ ] #1076 Coverage restructure: per-batch codecov uploads, delete the serial 2-h run
- [ ] #1061 HuggingFace hardening (collection-time download → fixture; HF cache; `HF_HUB_DOWNLOAD_TIMEOUT`; `HF_TOKEN`)
- [ ] #1077 Rebalance slow batches (after #1074, priced on post-fix durations)

**Phase 2 — test surgery (rigor-preserving, ~25–30 min off the suite)**
- [ ] #1078 Reduce the cloned 20-row dispatch grids (with a one-off coverage diff)
- [ ] #1079 Thinned fixture variant for `test_sample_posterior_predictive`
- [ ] #1080 Predictive groups in the cartoon fixtures
- [ ] #1081 Batch of small measured trims

**Phase 3 — exploratory (only after Phase 1 telemetry exists)**
- [ ] #1082 pytest-xdist pilot on one batch

## Expected end state

| Milestone | Slow suite (summed) | Batched wall | Release test gate |
|---|---|---|---|
| Today | ~96 min | 53 min (worst batch) | **times out** |
| After Phase 0 | ~96 min | 53 min | ~57 min, passes |
| After Phase 1 | ~85 min | ~30 min | ~33 min |
| After Phase 2 | ~55–60 min | ~20 min | ~22 min |
| Phase 3 (if xdist lands) | — | ~10–15 min | ~15 min |

Fast PR suite stays ~8 min throughout (already healthy).

## Ground rules for all child PRs

- One concern per PR; paste before/after `--durations` numbers in the description.
- Respect the rigor floor in #1078 — never prune the only real-sampling test of a dispatch path, the VI end-to-end regressions (#1056), the missing-data ONNX-under-NUTS tests, or the aDDM PPC tests.
- Any test-pruning PR attaches a codecov before/after (the coverage job runs the full suite, so pruning can move the tracked number).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

This is a tracking issue rather than a single implementation task. Start by reading tests/, .github/workflows/, and pyproject.toml, then choose one unclaimed child issue from phases 0–3. A child task is done when its stated change lands with before/after duration data and any required coverage evidence; the overall plan is complete when the release gate and suite meet the listed end-state targets.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, devops, performance, testing-qa
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.