tests: reduce the cloned 20-row sampler-dispatch grids to a coverage-preserving 9-row design
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 24
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 60
Description
Problem
The slow suite is dominated by a sampler-dispatch smoke matrix with heavy duplication (~43 min of the ~96-min suite; durations from run 29419740259, py3.12):
- The same 20-row
(loglik_kind × backend × sampler × step)grid is cloned intests/slow/test_mcmc.py:19-40,tests/slow/test_missing_data_mcmc.py:46-67, andtests/slow/test_missing_data_and_deadline_mcmc.py:66-88, crossed with 9 test functions (~140 cases;tests/slow/test_choice_only.pyadds a 9-row grid × 4 near-identical functions). Assertions are almost entirelyisinstance(idata, xr.DataTree). - Only ~9 of the 20 rows exercise distinct HSSM code paths:
step='slice'rows passpm.Slicethrough**kwargsstraight to PyMC — zero HSSM code (NUTS rows already evaluate logp+dlogp, a superset).- explicit
sampler='pymc'duplicates the default-resolution branch (src/hssm/base.py:655-662). - blackbox
(None,None),('pymc',None),('pymc','slice')are three near-identical Slice runs — HSSM injectspm.Sliceitself (base.py:670-671). - 5 rows per grid are pure argument-validation ValueErrors (
base.py:664-668,694-698) that never sample but pay full model construction (incl. ONNX load): 57 cases, 4.1 min total.
- Sampling settings are already minimal (
tune=10, draws=10, chains=1, cores=1) — per-test cost is model build + backend compilation, so the lever is grid size, not draw counts.
Plan
- In
test_mcmc.py, replace the 20-row grid with a 9-row covering set: analytical/default, analytical/numpyro, approx_differentiable-pytensor/default, ad-pytensor/numpyro, ad-jax/default (exercises default→numpyro resolution), ad-jax/pymc (the only JAX-op-under-PyMC-NUTS coverage), blackbox/default (auto-Slice injection), blackbox/numpyro→ValueError, analytical/numpyro+slice→ValueError. Keep one real slice row (e.g. analytical/pymc/slice) as PyMC-passthrough smoke. - Slim the two missing-data clones to ~4 rows (one per loglik_kind/backend build): their unique value is the missing-net wrapping under NUTS (
src/hssm/hssm.py:373-380), not sampler dispatch. - Consolidate the 3 validation rules into ~3 fast (non-slow)
pytest.raisesunit tests on a cheap analytical DDM. test_choice_only.py: run the full grid only ontest_choice_only_default_params; run the regression variants (beta / logit1 / combined) on the default row only. Keep both backend variants — jax wrap vs pytensor direct is real dispatch (src/hssm/distribution_utils/dist.py:804-816).- Keep the log-likelihood recompute equality checks (
test_mcmc.py:72-80,test_missing_data_mcmc.py:91-98) on every surviving sampled row — the only numeric end-to-end validation of that path. - Attach a before/after coverage diff to the PR: the coverage job runs the full suite, so pruning can move the tracked codecov number; expected delta ≈ 0, demonstrate it once (see #1076).
Rigor floor — do not prune or mock
- One real-sampling test per distinct dispatch path (the 9 rows above).
- End-to-end VI tests
tests/slow/test_vi.py:120-147(issue #1056 regressions). - Missing-data/deadline ONNX mixtures under actual NUTS.
- aDDM PPC conditioning/continuation tests.
Acceptance
- ~22–26 min off the summed slow suite (verify via
--durations=50). - Coverage diff shows ~no change.
- One shared grid constant (no more per-file clones drifting apart).
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the cloned grids in tests/slow/test_mcmc.py, tests/slow/test_missing_data_mcmc.py, tests/slow/test_missing_data_and_deadline_mcmc.py, and tests/slow/test_choice_only.py, then review the referenced dispatch and missing-data paths in src/hssm/base.py, src/hssm/hssm.py, and src/hssm/distribution_utils/dist.py. Preserve the listed sampling, validation, VI, missing-data, and PPC coverage while consolidating the grids and moving validation checks to fast tests. Run the full suite with --durations=50 and attach a before/after coverage diff.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100