support multiple configurations per regression test
Nobody has claimed this yet.
- Dominant language
- OCaml
- Stars
- 252
- Forks
- 90
- Avg merge
- 4d 1m
- Merged PRs (30d)
- 19
Description
We should have a way to check the same regression test with multiple configurations to avoid duplicating files with just different PARAM: ... lines at the top.
Would be good to run several solvers on the same test. E.g. https://github.com/goblint/analyzer/blob/master/tests/regression/34-localization/01-nested.c should now also be tested with td3 after https://github.com/goblint/analyzer/commit/cf11c4b0b3602f417a5573e56853b81940384998.
Also see https://github.com/goblint/analyzer/pull/196#issuecomment-850299611.
We could just have several lines starting with PARAM: or something like PARAM+: which appends/overrides the previous line.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/regression/34-localization/01-nested.c, the referenced commit, and the regression-test harness that interprets its PARAM lines. Read PR #196's linked discussion to understand the proposed configuration semantics. Done means one regression test can run with multiple solver configurations without duplicating the test file, including the td3 case described here.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, ocaml
- Domain
- testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100