Add software regression tests
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.8k
- Forks
- 388
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 15
Description
Summary:
We don't have software regression tests in place. We should introduce them and test them.
Description:
More accurately, we could use some characterization tests.
Here are a few things that we should test:
- reproducibility between runs with the same seed for some models (tbd)
- reproducibility across different commits with some models (tbd)
It's ok that things change over time. We just don't ever want the changes to happen unintentionally.
(There's no reason we can't characterize Stan's behavior before we agree on how to test correctness in light of stochastic algorithms. Once a random seed is set [with a particular set of hardware, environment, etc], it should run the same.)
Current Version:
v2.15.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating Stan's existing model test infrastructure, then define characterization cases for reproducibility with the same seed and across commits. Done means the agreed regression tests detect unintended behavioral changes while accounting for hardware and environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100