Invalid pointer cast generation when working with global variables
- Dominant language
- C++
- Stars
- 8.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
_Situation described below is only valid for binary files compiled with debug information._
Decompilation of a program that uses such types for global variables that are smaller than a size of an integer of the architecture where the program was compiled leads to a generation of an invalid cast in places where a pointer to these variables is needed.
For example simple program written in c and compiled with GCC/Clang:
```
char u = 0;
int main()
{
scanf("%c", &u);
printf("You typed %d\n", (int32_t)u);
return 0;
}
```
is decompiled to:
```
char u = 0;
int main(int argc, char ** argv) {
scanf("%c", (char *)(int32_t)(char)&u);
printf("You typed %d\n", (int32_t)u);
return 0;
}
```
As you can see in the expression `scanf("%c", (char *)(int32_t)(char)&u);` the pointer `&u` is firstly casted to the type of global variable `u`. This is incorrect because the type of the variable `u` is an 8bit char which means that if I compiled this result of decompilation on a 32bit machine with a 32bit address I would lose 24bits from the address.
As I wrote before, this situation only applies only to the binaries that are compiled with debug information.
I traced the source of the problem and this cast is generated by module `simple_types`. I tried to understand what this module does by looking at its code. I think that it tries to propagate type of the variable to all expressions that this variable is used in and if I am not wrong It does so by analyzing the type of expressions it is used in. If the type is from debug information then it has higher priority than any other found type and thus every expression is cast to the type.
I altered the code to hopefully act the same way it would act without debug information. The code can be found in my [fork](https://github.com/xkubov/retdec/tree/invalid-casts) on branch `invalid-casts`. I did not want to create pull request because I am not really sure what everything is module `simple_types` suppose to do and wanted someone to take a look at this issue first (I can do so afterward).
I did run all regression tests after alternation of code in `simple_types` and everything passed successfully. Also, I created a simple integration test with a few binaries of different architectures for repository `retdec-regression-tests` specially designed for this case. For this change, I will send a pull request as I find it to be independent of the issue described in here.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the simple_types module and the issue's example of global-variable pointer casts. Compare the proposed change in the linked invalid-casts branch with the existing regression tests, then run the full regression suite and the retdec-regression-tests integration case; done means generated pointers no longer pass through the global variable's narrow type while existing tests remain passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, cpp
- Domain
- compilers, reverse-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100