Ecdar / Ecdar/j-Ecdar

Possibly incorrect tests

Open
#79 1 comment 0 reactions 0 assignees View on GitHub
bug help wanted invalid question
Dominant language
Java
Stars
4
Forks
9
PR merge metrics
No merged PRs in 30d

Description

There are currently 7 ignored tests.

[testRefinementByNiels](https://github.com/Ecdar/j-Ecdar/blob/984419562b7e4ef693abc6ea883edb9674e57527/test/features/BoolTest.java#L533): Is non-deterministic. The CI nearly always have this test pass, but on my local development PCs it fail 90% of the time. The assertions which fails are ``assert(new SimpleTransitionSystem(auts[1]).isFullyConsistent());`` and ``assert(new SimpleTransitionSystem(auts[1]).isLeastConsistent());``. When removing the ``assert(new SimpleTransitionSystem(auts[1]).isDeterministic());`` they still fail meaning that the ``isDeterminsitic`` check does not alter the state of the automaton.
[Z2RefinesZ2Z3Z4](https://github.com/Ecdar/j-Ecdar/blob/984419562b7e4ef693abc6ea883edb9674e57527/test/features/DelayRefinementTest.java#L324): ``Z2 <= Z2 \\ Z3``. Fails refinement because ``Z2`` has output ``o``and ``Z2 \\ Z3`` has the same as input. Both ``Z2`` and ``Z3`` have no inputs and only ``o`` as an output. As the quotient inputs are ``Act_i^T ∪ Act_o^S`` ``o`` will always be an input of the quotient. Either this test is incorrect or the channel ``o`` should be distinct between ``Z2`` and ``Z3``.
[testBoolSafeLoadXML](https://github.com/Ecdar/j-Ecdar/blob/984419562b7e4ef693abc6ea883edb9674e57527/test/features/BoolTest.java#L612): Relies on files from the ``testOutput`` directory which cannot be found when running the tests.
[T0RefinesT3T1T2](https://github.com/Ecdar/j-Ecdar/blob/984419562b7e4ef693abc6ea883edb9674e57527/test/features/DelayRefinementTest.java#L416): Fails on two refinements ``(T3 \\ T4) \\ T2 <= (T3 \\ T4) \\ T2`` and ``T1 <= (T3 \\ T4) \\ T2``. The first assertion fails because of ``Output violation`` and the second because of a ``Delay violation``.
[newQuotientTest5](https://github.com/Ecdar/j-Ecdar/blob/984419562b7e4ef693abc6ea883edb9674e57527/test/features/UniversitySimpleTest.java#L172): Fails because the file ``comp.xml`` cannot be found.
[testFromTestFramework1](https://github.com/Ecdar/j-Ecdar/blob/984419562b7e4ef693abc6ea883edb9674e57527/test/features/UniversityTest.java#L245): Fails refinement on ``Machine <= ((((Adm2 && HalfAdm1) || Machine || Researcher) \\ (Adm2 && HalfAdm2)) \\ Researcher)`` but a test passes on ``Machine <= (Machine || ((Adm2 && HalfAdm1) || Researcher)) \\ ((Adm2 && HalfAdm2) || Researcher)``.
[testFromTestFramework2](https://github.com/Ecdar/j-Ecdar/blob/984419562b7e4ef693abc6ea883edb9674e57527/test/features/UniversityTest.java#L275): Fails refinement on ``Administration <= (Spec \\ Machine) \\ Researcher`` but a test passes on ``Administration <= (Spec \\ (Machine || Researcher))``.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the ignored tests listed in test/features/BoolTest.java, DelayRefinementTest.java, UniversitySimpleTest.java, and UniversityTest.java, then compare the reported failures with the referenced assertions and missing testOutput or comp.xml files. Determine for each test whether the test, fixture, or implementation is incorrect, and finish with the seven tests no longer incorrectly ignored or with their expected behavior documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.