avast / avast/retdec

Stack variable optimization

Open
#212 0 comments 0 reactions 0 assignees View on GitHub
C-bin2llvmir enhancement
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

Currently, all the reconstructed stack variables are protected from LLVM optimization passes by the `-stack-protect` pass. It adds to IR artificial variable uses that cannot be removed, and therefore variables themselves cannot be optimized away.

**Advantages**
- We want to prefer these stack variables to other types of LLVM IR objects. They have often additional metadata (e.g. names from debug info), and are more significant than temporary variables. However, LLVM does not know this. If they are not protected, it often optimizes them away in favor of less significant objects.
- Our interprocedural stack data-flow analysis is poor/non-existent. But stack is shared between functions. One function can write there something and never use it (LLVM would remove such variable), other function can read this later.
- There might be some shady malware stuff going on that LLVM optimizations cannot handle.
- If we do not allow optimizations, we cannot screw it up.

**Disadvantages**
- Obviously, this produces suboptimal, often very ugly, code.
- _Llvmir2hll_ is still doing stack optimizations. These would be performed faster in _bin2llvmir_.

**Solution**
I don't really know. These things go against each other. Find some compromise. See what would Hex-Rays do in some of these cases.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by examining the `-stack-protect` pass and how `llvmir2hll` and `bin2llvmir` handle stack optimizations. Compare the stated preservation needs with the code-quality disadvantages, then investigate a compromise; done requires an agreed approach that balances both concerns.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, reverse-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.