lnccbrown / lnccbrown/HSSM

tests: pytest addopts cleanup — drop global --cov/--exitfirst/--capture=no, add --durations, raise --timeout

Open
#1,075 1 comment 0 reactions 0 assignees View on GitHub
chore
Dominant language
Python
Stars
124
Forks
24
Avg merge
19h 32m
Merged PRs (30d)
60

Description

## Problem

The `[tool.pytest.ini_options]` addopts in `pyproject.toml` apply to **every** pytest invocation (PR fast jobs, slow batches, release gate, coverage job), and several are costing time or hiding information:

- `--cov=. --cov-report=term-missing --cov-report=xml` — instruments every job, but the coverage produced by PR/slow/release jobs is never uploaded anywhere (only `coverage.yml` uploads). Measured overhead: up to ~2.5× on Python-heavy code with the default tracer (+21% with `COVERAGE_CORE=sysmon`). Also `--cov=.` silently **unions** with `coverage.yml`'s CLI `--cov=src/hssm`, defeating its scoping.
- `--exitfirst` — combined with 20–55 min CI iterations, this yields one-failure-per-push debugging; in the coverage job a single late failure aborts a ~2 h run with nothing uploaded.
- `--capture=no` + `log_cli = INFO` — a log firehose that is currently the *only* per-test timing source, because no `--durations` is configured (this analysis had to scrape GH Actions log timestamps).
- `--timeout=360` — signal-based: it does not guard collection (#1061) and it converted the suite's slowest test into a guaranteed timeout+untimed-rerun double-run (see the waic_loo issue).
- `--reruns=2` — masks real flakes; worst case triples a failing slow test.

## Plan

- Replace addopts with: `--durations=50 --durations-min=1 --timeout=600 --reruns=1 --reruns-delay=5`.
- Move coverage flags out of addopts; `coverage.yml` already passes `--cov=src/hssm --cov-report=xml` on its own CLI (verify post-removal; also set `COVERAGE_CORE=sysmon` there).
- Drop `--exitfirst` (use `-x` locally when wanted).
- Drop `--capture=no`; set `log_cli_level = "WARNING"`. If hang forensics are still wanted on slow batches, pass `-o log_cli_level=INFO` only on the slow-path pytest invocation in `run_tests.yml`.
- Fix stale docs in the same PR: the `--runslow` flag referenced in the `slow` marker help text (`pyproject.toml`) and in `CLAUDE.md` **does not exist** (there is no `pytest_addoption` hook — plain `pytest` runs everything, which is also why the coverage job silently runs the full suite). Either implement the flag or correct both references to `-m "not slow"` / `-m slow`.

## Acceptance

- `--durations=50` tables appear in all CI jobs (future audits become free).
- Fast PR job and slow batches measurably faster (record before/after in the PR).
- `coverage.yml` still uploads a valid `coverage.xml` scoped to `src/hssm`.
- No stale `--runslow` references remain.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Research direction

Start with the pytest options in pyproject.toml, then inspect coverage.yml, run_tests.yml, and the --runslow references in CLAUDE.md. Verify the coverage command remains scoped to src/hssm, choose whether to implement or correct the stale flag references, and compare CI timings before and after. Done means all CI jobs show duration tables, coverage.xml uploads successfully, and no stale --runslow references remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, documentation, testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.