lnccbrown / lnccbrown/HSSM

tests: reduce the cloned 20-row sampler-dispatch grids to a coverage-preserving 9-row design

Open
#1,078 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

chore refactor
Dominant language
Python
Stars
124
Forks
24
Avg merge
19h 32m
Merged PRs (30d)
60

Description

Problem

The slow suite is dominated by a sampler-dispatch smoke matrix with heavy duplication (~43 min of the ~96-min suite; durations from run 29419740259, py3.12):

  • The same 20-row (loglik_kind × backend × sampler × step) grid is cloned in tests/slow/test_mcmc.py:19-40, tests/slow/test_missing_data_mcmc.py:46-67, and tests/slow/test_missing_data_and_deadline_mcmc.py:66-88, crossed with 9 test functions (~140 cases; tests/slow/test_choice_only.py adds a 9-row grid × 4 near-identical functions). Assertions are almost entirely isinstance(idata, xr.DataTree).
  • Only ~9 of the 20 rows exercise distinct HSSM code paths:
    • step='slice' rows pass pm.Slice through **kwargs straight to PyMC — zero HSSM code (NUTS rows already evaluate logp+dlogp, a superset).
    • explicit sampler='pymc' duplicates the default-resolution branch (src/hssm/base.py:655-662).
    • blackbox (None,None), ('pymc',None), ('pymc','slice') are three near-identical Slice runs — HSSM injects pm.Slice itself (base.py:670-671).
    • 5 rows per grid are pure argument-validation ValueErrors (base.py:664-668, 694-698) that never sample but pay full model construction (incl. ONNX load): 57 cases, 4.1 min total.
  • Sampling settings are already minimal (tune=10, draws=10, chains=1, cores=1) — per-test cost is model build + backend compilation, so the lever is grid size, not draw counts.

Plan

  1. In test_mcmc.py, replace the 20-row grid with a 9-row covering set: analytical/default, analytical/numpyro, approx_differentiable-pytensor/default, ad-pytensor/numpyro, ad-jax/default (exercises default→numpyro resolution), ad-jax/pymc (the only JAX-op-under-PyMC-NUTS coverage), blackbox/default (auto-Slice injection), blackbox/numpyro→ValueError, analytical/numpyro+slice→ValueError. Keep one real slice row (e.g. analytical/pymc/slice) as PyMC-passthrough smoke.
  2. Slim the two missing-data clones to ~4 rows (one per loglik_kind/backend build): their unique value is the missing-net wrapping under NUTS (src/hssm/hssm.py:373-380), not sampler dispatch.
  3. Consolidate the 3 validation rules into ~3 fast (non-slow) pytest.raises unit tests on a cheap analytical DDM.
  4. test_choice_only.py: run the full grid only on test_choice_only_default_params; run the regression variants (beta / logit1 / combined) on the default row only. Keep both backend variants — jax wrap vs pytensor direct is real dispatch (src/hssm/distribution_utils/dist.py:804-816).
  5. Keep the log-likelihood recompute equality checks (test_mcmc.py:72-80, test_missing_data_mcmc.py:91-98) on every surviving sampled row — the only numeric end-to-end validation of that path.
  6. Attach a before/after coverage diff to the PR: the coverage job runs the full suite, so pruning can move the tracked codecov number; expected delta ≈ 0, demonstrate it once (see #1076).

Rigor floor — do not prune or mock

  • One real-sampling test per distinct dispatch path (the 9 rows above).
  • End-to-end VI tests tests/slow/test_vi.py:120-147 (issue #1056 regressions).
  • Missing-data/deadline ONNX mixtures under actual NUTS.
  • aDDM PPC conditioning/continuation tests.

Acceptance

  • ~22–26 min off the summed slow suite (verify via --durations=50).
  • Coverage diff shows ~no change.
  • One shared grid constant (no more per-file clones drifting apart).

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the cloned grids in tests/slow/test_mcmc.py, tests/slow/test_missing_data_mcmc.py, tests/slow/test_missing_data_and_deadline_mcmc.py, and tests/slow/test_choice_only.py, then review the referenced dispatch and missing-data paths in src/hssm/base.py, src/hssm/hssm.py, and src/hssm/distribution_utils/dist.py. Preserve the listed sampling, validation, VI, missing-data, and PPC coverage while consolidating the grids and moving validation checks to fast tests. Run the full suite with --durations=50 and attach a before/after coverage diff.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.