avast / avast/retdec

Use calling conventions and stack traces in function parameter analysis

Open
#653 1 comment 0 reactions 1 assignee Claimed by @xkubov View on GitHub
C-bin2llvmir enhancement
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

Due to #652, `bugs.phase-debug-info-analysis-crash.Test (tail.ex --pdb tail.pdb -k)` failed. It was disabled in https://github.com/avast/retdec-regression-tests/commit/ad6389011753aee785e299f729c641f1b3e2507a.

The problem was present even before #652, but localizing registers made it visible - tests fail.

Test fail problem:
* Test for presence of string: `"Chyba: %s\n"`
* Code before #652:
```cpp
char * v1 = "Chyba: %s\n";
register = (int32_t)&v1;
// register never used in function
```
* After #652, register assignment as well as `v1` assignment are optimized away, because values are never used in the function.
* Missing string we tested for -> test failed.

Real problem:
* The string is not used, and therefore is missing (optimized away).
* It was not used before as well, but was preserved due to inter-procedural register analysis.
* The string should be used as an argument in `fprintf()` call.
* It is not detected as `fprint()` argument, because function parameter analysis halts tracking of possible function arguments at the first previous function calls. But in this case, parameter is stored to stack several calls before the call that uses the parameter is made:
```asm
.text:0040103B push offset Format ; "Chyba: %s\n" ; string to stack
.text:00401040 mov edi, esp
.text:00401042 call ds:__iob_func ; call NOT using the string
.text:00401048 cmp edi, esp
.text:0040104A call __RTC_CheckEsp ; call NOT using the string
; fprintf() arg tracking stops here
.text:0040104F add eax, 40h ; '@'
.text:00401052 push eax
.text:00401053 call ds:fprintf ; call using the string
```

Design and implement more robust function parameter analysis that could deal with this. It should track beyond previous function calls and make use of known calling conventions. A better stack analysis (#651) will also probably help a lot - or maybe even entirely solves the problem.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.