Regression tests on macOS sometimes randomly fail
- Dominant language
- C++
- Stars
- 8.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
[This macOS build](https://retdec-tc.avast.com/viewLog.html?buildId=3418&buildTypeId=Retdec_MacBuild) has succeeded. The one [after that](https://retdec-tc.avast.com/viewLog.html?buildId=3425&buildTypeId=Retdec_MacBuild), which is for a commit that only changed the changelog file, failed because of the following test:
```
bugs.selective-decompilation-strange-behaviour.TestDecompileSelect0x0DecodeAll (main.ex --sel[..]) [ FAIL ] (1.15s)
F
======================================================================
FAIL: test_has_no_functions (bugs.selective-decompilation-strange-behaviour.TestDecompileSelect0x0DecodeAll)
----------------------------------------------------------------------
Traceback (most recent call last):
File "retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py", line 65, in setUp
self.assertEqual(self.decompiler.return_code, 0, msg=msg)
AssertionError: 1 != 0 : decompiler failed; output:
...
Running phase: Simple types recovery optimization ( 0.65s )
Running phase: Stack pointer operations optimization ( 0.65s )
Running phase: Instruction idioms optimization ( 0.65s )
Running phase: LLVM ( 0.65s )
Running phase: LLVM instruction optimization ( 0.65s )
Running phase: Instruction idioms optimization ( 0.65s )
Running phase: Phi removal ( 0.65s )
Running phase: Value protection optimization ( 0.65s )
Error: Decompilation to LLVM IR failed
```
I [restarted the build](https://retdec-tc.avast.com/viewLog.html?buildId=3430&buildTypeId=Retdec_MacBuild&tab=buildLog) after that, and it failed again, but because of another test:
```
integration.palindrome.Test_2017 (palindrome.x86.clang.O0.g.elf) [ FAIL ] (0.85s)
F
======================================================================
FAIL: test_produce_expected_output (integration.palindrome.Test_2017)
----------------------------------------------------------------------
Traceback (most recent call last):
File "retdec-regression-tests/integration/palindrome/test.py", line 13, in test_produce_expected_output
expected_output='krk is a palindrome\n'
File "retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py", line 97, in assert_c_produces_output_when_run
self.assertEqual(actual_output, expected_output, msg)
AssertionError: 'Try again! (krk)\n' != 'krk is a palindrome\n'
- Try again! (krk)
+ krk is a palindrome
: where the input was: krk
```
We either have issues with non-determinism in our tests on macOS, or there is something wrong with the macOS agent.
Possibly related issues: #267, #265, #264.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked macOS build logs and the named regression tests in retdec-regression-tests/integration/palindrome/test.py and retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py. Re-run the failing tests and compare repeated builds, then review related issues #267, #265, and #264; done means the source of the macOS-only failures is isolated and repeat builds are reliable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- macos, python
- Domain
- ci-cd, operating-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100