lnccbrown / lnccbrown/HSSM

CI: pilot pytest-xdist on one slow batch

Open
#1,082 1 comment 0 reactions 0 assignees View on GitHub
enhancement github_actions
Dominant language
Python
Stars
124
Forks
24
Avg merge
19h 32m
Merged PRs (30d)
60

Description

## Opportunity

Slow tests are single-core by construction (`cores=1, chains=1, tune=10, draws=10`), so 3 of the runner's 4 vCPUs idle through every slow batch. `pytest-xdist` is already installed but unused. Potential: 2–3× per batch (a balanced ~30-min batch → ~12–16 min), the single largest remaining speedup after batching/rebalancing.

## Known risks (why this is a pilot, not a rollout)

- **Memory:** RSS grows ~0.6 → 4.9 GiB across one serial batch (native JAX/pytensor allocations; measured via the `HSSM_TEST_RSS_LOG` fixture in `tests/conftest.py`). N workers multiply resident sets on a 16 GB runner.
- `log_cli` / capture interactions garble under xdist.
- `pytest-rerunfailures` + xdist have known quirks.
- Per-worker pytensor/JAX compilation may eat into the win.

## Plan

1. Prerequisites: `--durations` telemetry and the batch rebalance must be landed first (see the addopts-cleanup and rebalance issues).
2. Add **one** matrix entry (single batch, single Python) running `-n 4 --dist loadfile`, with `HSSM_TEST_RSS_LOG=1`.
3. Observe for ~a week of main pushes: wall-clock, peak RSS per worker, flake/rerun rate vs. the serial twin batch.
4. Decide and record on this issue: roll out to all batches / reduce to `-n 2` / abandon.

## Acceptance

- A data-backed go/no-go decision with the measurements attached, not a silent adoption.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with tests/conftest.py to understand HSSM_TEST_RSS_LOG, then inspect the existing CI matrix and serial slow-batch configuration. Add the single specified -n 4 --dist loadfile pilot with RSS logging, compare it with the serial twin for about a week, and attach wall-clock, peak RSS, and flake/rerun measurements before recording a go/no-go decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, performance, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.