Stack optimization missing on create some stack variables
- Dominant language
- C++
- Stars
- 8.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
When I using retdec to reverse md5 algorithm. I found a kind of conditon that stack pass will fail to create stack variables. such as
```
store i32 %506, i32* %512
-----------------------------------------------
>| %511 = add i64 %510, %509
>| %510 = add i64 %507, -128
>| %507 = load i64, i64* @rbp
>| i64 -8
>| i64 -128
>| %509 = mul i64 %508, 4
>| %508 = load i64, i64* @rdx
>| %504 = load i64, i64* @rcx
>| %502 = load i32, i32* %stack_var_-156
>| %stack_var_-156 = alloca i32
>| i64 4
-----------------------------------------------
after simplifyNode
-----------------------------------------------
>| %511 = add i64 %510, %509
>| %509 = mul i64 %508, 4
>| %stack_var_-156 = alloca i32
>| i64 4
>| i64 -136
-----------------------------------------------
```
Obviously, %510 should be stack_var_-136. But stack pass will fail to recognize it, and will not create related stack variable.
Here is my binary. And this instruction is at address 0x400858
[md5.zip](https://github.com/avast/retdec/files/4392142/md5.zip)
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with the attached md5.zip binary at instruction address 0x400858 and inspect the stack pass output before and after simplifyNode. Trace why %510 is not recognized as stack_var_-136 and verify that the related stack variable is created and the shown expression is simplified correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, reverse-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100