scipp / scipp/copier_template

Automated regression tests

Open
#187 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Jinja
Stars
2
Forks
2
PR merge metrics
No merged PRs in 30d

Description

We need a way to test whether our workflows still produce the accepted 'correct' results after we make some changes. E.g. in https://github.com/scipp/esssans/pull/135 and https://github.com/scipp/esssans/pull/143. However, there are changes that should change the result, such as adding a new correction or tuning a parameter. In Mantid, those accepted results are written to file and loaded to compare them to results from a new version of the code. This needs extra infrastructure to store and provide the files and extra work to update them. Here is a potential alternative.

Have a test script that does this procedure on each PR:

  1. Check out main.
  2. Run tests with a specific mark and save results of those tests to a folder results_main.
  3. Check out the head of the PR branch.
  4. Run tests with the same mark and save results to results_branch.
  5. For each file that exists in both results_main and results_branch, load the file and compare with sc.testing.assert_identical and sc.testing.assert_allclose.

The tests run this way can contain assertions to, e.g., make sure that the result has the expected shape. But the main purpose of these tests is writing data. That data can be any scipp object, e.g., the result of running a workflow.

This procedure would perform regression tests against main which we assume has the accepted 'correct' code. But it does not require storing result files in a public location.

What do you think? Does this make sense?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the proposed procedure and the referenced PRs 135 and 143. Define how a test script would check out main and the PR branch, save marked-test outputs in results_main and results_branch, and compare shared files with sc.testing.assert_identical and sc.testing.assert_allclose. Done means the regression-test workflow and its result-comparison behavior are specified and implemented.

Written by the indexing model from the issue text.

Assessment

Domain
ci-cd, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.