avast / avast/retdec

Casting pointers to integers - better pointer data type propagation

Open
#213 0 comments 0 reactions 0 assignees View on GitHub
C-bin2llvmir enhancement P-output
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

In the decompiled code, there are often constructions like this:
Original code:
```c
size_t my_strlen(const char * str) {
const char *s;
for (s = str; *s; ++s);
return(s - str);
}
```
Decompiled code:
```c
int32_t my_strlen(char * str) {
int32_t v1 = (int32_t)str;
int32_t v2 = v1;
while (*(char *)v2 != 0) {
v2++;
}
return v2 - v1;
}
```
It is not ideal, pointers should be propagated better. The current code works only on 32-bit systems - if we want to recompile it (as we do in regression tests), option `-m32` has to be used. Otherwise, we would be casting 64-bit pointers to `int32_t`, which clearly would not work.

We will not be able to solve this 100%, since this is how it works in assembly and our analyses can not be always perfect. But we should try:
- Better type propagation - some of those variables could be pointers.
- Simple patterns identification and optimization - e.g. detect chain `pointer -> integer -> pointer`, and get rid of these casts.
- etc.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not name implementation files, tests, or an entry point. Start by reproducing the shown pointer-to-integer output and tracing the existing type propagation; done means pointer types are preserved or redundant pointer–integer–pointer casts are removed without breaking regression recompilation on 64-bit systems.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, cpp
Domain
compilers, reverse-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.