avast / avast/retdec

Regression tests on macOS sometimes randomly fail

Open
#702 0 comments 0 reactions 0 assignees View on GitHub
bug C-regression-tests O-macos P-output
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

[This macOS build](https://retdec-tc.avast.com/viewLog.html?buildId=3418&buildTypeId=Retdec_MacBuild) has succeeded. The one [after that](https://retdec-tc.avast.com/viewLog.html?buildId=3425&buildTypeId=Retdec_MacBuild), which is for a commit that only changed the changelog file, failed because of the following test:
```
bugs.selective-decompilation-strange-behaviour.TestDecompileSelect0x0DecodeAll (main.ex --sel[..]) [ FAIL ] (1.15s)
F
======================================================================
FAIL: test_has_no_functions (bugs.selective-decompilation-strange-behaviour.TestDecompileSelect0x0DecodeAll)
----------------------------------------------------------------------
Traceback (most recent call last):
File "retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py", line 65, in setUp
self.assertEqual(self.decompiler.return_code, 0, msg=msg)
AssertionError: 1 != 0 : decompiler failed; output:
...
Running phase: Simple types recovery optimization ( 0.65s )
Running phase: Stack pointer operations optimization ( 0.65s )
Running phase: Instruction idioms optimization ( 0.65s )
Running phase: LLVM ( 0.65s )
Running phase: LLVM instruction optimization ( 0.65s )
Running phase: Instruction idioms optimization ( 0.65s )
Running phase: Phi removal ( 0.65s )
Running phase: Value protection optimization ( 0.65s )
Error: Decompilation to LLVM IR failed
```
I [restarted the build](https://retdec-tc.avast.com/viewLog.html?buildId=3430&buildTypeId=Retdec_MacBuild&tab=buildLog) after that, and it failed again, but because of another test:
```
integration.palindrome.Test_2017 (palindrome.x86.clang.O0.g.elf) [ FAIL ] (0.85s)
F
======================================================================
FAIL: test_produce_expected_output (integration.palindrome.Test_2017)
----------------------------------------------------------------------
Traceback (most recent call last):
File "retdec-regression-tests/integration/palindrome/test.py", line 13, in test_produce_expected_output
expected_output='krk is a palindrome\n'
File "retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py", line 97, in assert_c_produces_output_when_run
self.assertEqual(actual_output, expected_output, msg)
AssertionError: 'Try again! (krk)\n' != 'krk is a palindrome\n'
- Try again! (krk)
+ krk is a palindrome
: where the input was: krk
```
We either have issues with non-determinism in our tests on macOS, or there is something wrong with the macOS agent.

Possibly related issues: #267, #265, #264.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked macOS build logs and the named regression tests in retdec-regression-tests/integration/palindrome/test.py and retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py. Re-run the failing tests and compare repeated builds, then review related issues #267, #265, and #264; done means the source of the macOS-only failures is isolated and repeat builds are reliable.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, python
Domain
ci-cd, operating-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.