avast / avast/retdec

Stack optimization missing on create some stack variables

Open
#727 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

When I using retdec to reverse md5 algorithm. I found a kind of conditon that stack pass will fail to create stack variables. such as
```
store i32 %506, i32* %512
-----------------------------------------------
>| %511 = add i64 %510, %509
>| %510 = add i64 %507, -128
>| %507 = load i64, i64* @rbp
>| i64 -8
>| i64 -128
>| %509 = mul i64 %508, 4
>| %508 = load i64, i64* @rdx
>| %504 = load i64, i64* @rcx
>| %502 = load i32, i32* %stack_var_-156
>| %stack_var_-156 = alloca i32
>| i64 4
-----------------------------------------------
after simplifyNode
-----------------------------------------------
>| %511 = add i64 %510, %509
>| %509 = mul i64 %508, 4
>| %stack_var_-156 = alloca i32
>| i64 4
>| i64 -136
-----------------------------------------------
```
Obviously, %510 should be stack_var_-136. But stack pass will fail to recognize it, and will not create related stack variable.

Here is my binary. And this instruction is at address 0x400858
[md5.zip](https://github.com/avast/retdec/files/4392142/md5.zip)

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with the attached md5.zip binary at instruction address 0x400858 and inspect the stack pass output before and after simplifyNode. Trace why %510 is not recognized as stack_var_-136 and verify that the related stack variable is created and the shown expression is simplified correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, reverse-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.