runtimeverification / runtimeverification/kontrol
CSE tests failing in booster performance script
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 122
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
We use the following script https://github.com/runtimeverification/hs-backend-booster/blob/main/scripts/performance-tests-kontrol.sh to test for any regressions in kontrol w.r.t changes introduced in the booster. Recently, we have been getting multiple failed tests, namely:
FAILED src/tests/integration/test_kontrol_cse.py::test_foundry_dependency_automated[ArithmeticCallTest.test_double_add(uint256,uint256)]
FAILED src/tests/integration/test_kontrol_cse.py::test_foundry_dependency_automated[ArithmeticCallTest.test_double_add_double_sub(uint256,uint256)]
FAILED src/tests/integration/test_kontrol_cse.py::test_foundry_dependency_automated[ArithmeticCallTest.test_double_add_sub_external(uint256,uint256,uint256)]
FAILED src/tests/integration/test_kontrol_cse.py::test_foundry_dependency_automated[Identity.applyOp(uint256)]
from discussion with @PetarMax it seems these test may share state/have race conditions on file write/access, which seems consistent with our experience because running these tests separately doesn't seem to show any issues.
To reproduce this issue, cd into the booster repository and run scripts/performance-tests-kontrol.sh. To run the tests on a branch of kontrol different to master, use KONTROL_VERSION=feature/branch scripts/performance-tests-kontrol.sh
log from a failing run: kontrol-v0.1.222-sam-apply-equations-refactor.log
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run scripts/performance-tests-kontrol.sh from the booster repository, including the listed KONTROL_VERSION variant when relevant. Start with src/tests/integration/test_kontrol_cse.py and compare the four failing tests with separate runs; done means the performance script runs without these failures or race-related file-access errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash, python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100