Stack variable optimization
- Dominant language
- C++
- Stars
- 8.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
Currently, all the reconstructed stack variables are protected from LLVM optimization passes by the `-stack-protect` pass. It adds to IR artificial variable uses that cannot be removed, and therefore variables themselves cannot be optimized away.
**Advantages**
- We want to prefer these stack variables to other types of LLVM IR objects. They have often additional metadata (e.g. names from debug info), and are more significant than temporary variables. However, LLVM does not know this. If they are not protected, it often optimizes them away in favor of less significant objects.
- Our interprocedural stack data-flow analysis is poor/non-existent. But stack is shared between functions. One function can write there something and never use it (LLVM would remove such variable), other function can read this later.
- There might be some shady malware stuff going on that LLVM optimizations cannot handle.
- If we do not allow optimizations, we cannot screw it up.
**Disadvantages**
- Obviously, this produces suboptimal, often very ugly, code.
- _Llvmir2hll_ is still doing stack optimizations. These would be performed faster in _bin2llvmir_.
**Solution**
I don't really know. These things go against each other. Find some compromise. See what would Hex-Rays do in some of these cases.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by examining the `-stack-protect` pass and how `llvmir2hll` and `bin2llvmir` handle stack optimizations. Compare the stated preservation needs with the code-quality disadvantages, then investigate a compromise; done requires an agreed approach that balances both concerns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, reverse-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100