runtimeverification / runtimeverification/mir-semantics

Test runner connecting to non-existent external haskell backend

Open
#588 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
52
Forks
5
PR merge metrics
No merged PRs in 30d

Description

When reviewing the stable-mir-json update PR I ran into an issue with the haskell backend not being found.

Steps I took:

git worktree add ../mir-semantics-review
cd ../mir-semantics-review
git switch _update-deps/runtimeverification/stable-mir-json
git submodule update --recursive
make update-smir-tests

The output is below in the details tab

test_case = ('Array-index-writes', PosixPath('/home/zncv/Applications/mir-semantics-review/kmir/src/tests/integration/data/exec-sm...home/zncv/Applications/mir-semantics-review/kmir/src/tests/integration/data/exec-smir/arrays/array_write.state'), None)
kmir_backend = <kmir.kmir.KMIR object at 0x7f178150a960>, update_expected_output = True

    @pytest.mark.parametrize('kmir_backend', [KMIR(LLVM_DEF_DIR), KMIR(HASKELL_DEF_DIR)], ids=['llvm', 'haskell'])
    @pytest.mark.parametrize(
        'test_case',
        EXEC_DATA,
        ids=[name for (name, _, _, _) in EXEC_DATA],
    )
    def test_exec_smir(
        test_case: tuple[str, Path, Path, int],
        kmir_backend: KMIR,
        update_expected_output: bool,
    ) -> None:
    
        (_, input_json, output_kast, depth) = test_case
    
        parser = Parser(kmir_backend.definition)
    
        with input_json.open('r') as f:
            json_data = json.load(f)
        parsed = parser.parse_mir_json(json_data, 'Pgm')
        assert parsed is not None
        kmir_kast, _ = parsed
    
>       result = kmir_backend.run_parsed(kmir_kast, depth=depth)

src/tests/integration/test_integration.py:381: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/kmir/kmir.py:79: in run_parsed
    result = self.run_pattern(init_kore, depth=depth)
.venv/lib/python3.12/site-packages/pyk/ktool/krun.py:293: in run_pattern
    res = parser.pattern()
.venv/lib/python3.12/site-packages/pyk/kore/parser.py:202: in pattern
    name = self._match(TokenType.ID)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyk.kore.parser.KoreParser object at 0x7f1780c72300>, token_type = <TokenType.ID: 11>

    def _match(self, token_type: TokenType) -> str:
        if self._la.type != token_type:
>           raise ValueError(f'Expected {token_type.name}, found: {self._la.type.name}')
E           ValueError: Expected ID, found: EOF

.venv/lib/python3.12/site-packages/pyk/kore/parser.py:133: ValueError
==================================================================== short test summary info ====================================================================
FAILED src/tests/integration/test_integration.py::test_exec_smir[main-a-b-c-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[main-a-b-c --depth 19-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[call-with-args-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[assign-cast-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[structs-tuples-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[struct-field-update-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[arithmetic-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[arithmetic-unchecked-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[unary-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Ref-simple-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Ref-refAsArg-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Ref-refAsArg2-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Ref-refReturned-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Ref-doubleRef-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Ref-mutableRef-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Ref-weirdRefs-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[enum-discriminants-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Array-indexing-haskell] - ValueError: Expected ID, found: EOF
FAILED src/tests/integration/test_integration.py::test_exec_smir[Array-index-writes-haskell] - ValueError: Expected ID, found: EOF
======================================================== 19 failed, 19 passed, 105 deselected in 39.32s =========================================================
make: *** [Makefile:115: update-exec-smir] Error 1

This was solved by:

make build
make update-smir-tests

So I guess the test-runner is not creating or connecting to its own haskell kdist?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Makefile target update-exec-smir, then read src/tests/integration/test_integration.py around test_exec_smir and src/kmir/kmir.py around run_parsed. Reproduce the reported make update-smir-tests failure without make build, and determine whether the Haskell backend is created or connected before the integration tests run. Done means the update command handles the backend setup consistently and the Haskell cases pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
haskell, python
Domain
backend, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.