avast / avast/retdec

Some loops are erroneously turned into infinite loops

Open
#538 1 comment 0 reactions 0 assignees View on GitHub
bug C-bin2llvmir P-output
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

Hi,

When this kind of source code (loop with a small body) is compiled in 32-bit with gcc, RetDec fails to properly decompile it. In fact, this source code :

~~~C
int myfunc(unsigned char* S) {
for(int i = 0; i < 256; i++)
S[i] = i;
printf("Supposed not to be removed.");
return 0;
}
~~~
becomes :
~~~C
int32_t myfunc(int32_t * a1) {
__x86_get_pc_thunk_ax(g2, 0);
char v1 = 0;
while (true) {
*(char *)((int32_t)v1 + (int32_t)a1) = v1;
v1++;
}
}
~~~
RetDec properly decompiles it when it's compiled in 64-bit though :
~~~C
int64_t myfunc(int64_t * a1) {
int32_t v1 = 0;
for (int64_t i = 0; i < 256; i++) {
*(char *)(i + (int64_t)a1) = (char)v1;
v1++;
}
printf("Supposed not to be removed.");
return 0;
}
~~~

Three passes seem to cause this issue : *Instruction Combine* (-instcombine), *Global Value Numbering* (-gvn) and *Early Cse* (-early-cse). In that case they tend to transform a "br \ \ \" into a "br false \ \", as my IR dumps show, resulting in a "while true". The code that follow is then considered unreachable and removed. When I disable these passes in RetDec, the decompilation outputs the expected C code that is just like the 64-bit example above.

I also tried with llvm-8 (on your new branch), doesn't work either.
This is probably due to an upstream problem.

However, this problem doesn't occur when the binary is compiled with clang.

Note : This is not gcc's fault since the generated assembly doesn't represent an infinite loop, and the binary is working as expected.

Thank you for your time !

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the 32-bit GCC case and inspect the IR around the named Instruction Combine, Global Value Numbering, and Early CSE passes. Done means the decompiler preserves the finite loop and does not remove the following printf and return; the payload names no source files or tests to inspect.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, cpp
Domain
compilers, reverse-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.