Automated regression tests
Nobody has claimed this yet.
- Dominant language
- Jinja
- Stars
- 2
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
We need a way to test whether our workflows still produce the accepted 'correct' results after we make some changes. E.g. in https://github.com/scipp/esssans/pull/135 and https://github.com/scipp/esssans/pull/143. However, there are changes that should change the result, such as adding a new correction or tuning a parameter. In Mantid, those accepted results are written to file and loaded to compare them to results from a new version of the code. This needs extra infrastructure to store and provide the files and extra work to update them. Here is a potential alternative.
Have a test script that does this procedure on each PR:
- Check out
main. - Run tests with a specific mark and save results of those tests to a folder
results_main. - Check out the head of the PR branch.
- Run tests with the same mark and save results to
results_branch. - For each file that exists in both
results_mainandresults_branch, load the file and compare withsc.testing.assert_identicalandsc.testing.assert_allclose.
The tests run this way can contain assertions to, e.g., make sure that the result has the expected shape. But the main purpose of these tests is writing data. That data can be any scipp object, e.g., the result of running a workflow.
This procedure would perform regression tests against main which we assume has the accepted 'correct' code. But it does not require storing result files in a public location.
What do you think? Does this make sense?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the proposed procedure and the referenced PRs 135 and 143. Define how a test script would check out main and the PR branch, save marked-test outputs in results_main and results_branch, and compare shared files with sc.testing.assert_identical and sc.testing.assert_allclose. Done means the regression-test workflow and its result-comparison behavior are specified and implemented.
Written by the indexing model from the issue text.
Assessment
- Domain
- ci-cd, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100