The problem with static analysis of VMP-like protectors. CODE && DATA detect. ROP problems
- Dominant language
- C++
- Stars
- 8.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
The problem lies in the obvious disadvantage of static analysis alone. I'll tell you everything now:
We take an example(MsgB) from VMP 2.13.2
Clear example: [prog1.zip](https://github.com/avast/retdec/files/11346990/prog1.zip)
VMP 2.13.2 example: [prog1.vmp.exe.zip](https://github.com/avast/retdec/files/11346992/prog1.vmp.exe.zip)
If you go into the example with VMP, you will see that the sections with variables are empty, and API calls are encrypted
VMP 2.13.2 example:
VMP 2.13.2 RDATA

VMP 2.13.2 DATA

VMP 2.13.2 RDATA API

Clear example:
Clear RDATA

Clear DATA

Clear RDATA API

;---END
It's contains both example in RDATA API
On the first line - MessageBoxA
On the second line - ExitProcess
Also, do not pay attention to the inscription in MsgB, first I packed VMP 3.8.1, but I thought it was better to start with something old
As you can see in the case of VMP 2.13.2, the VMP loader deletes everything, and it is impossible to perform static analysis of the virtualized or mutated EntryPoint function, since our OEP is essentially virtualized/mutated. We can't even dump and start the analysis in static, since the OEP is virtualized/mutated. I'm still thinking about solving this problem. If anyone has any ideas, I will be glad if you can share it
p.s.
When trying to deobfuscate the example with VMP, LLVM deletes the entire code (it does the right thing). LLVM, or rather its optimization phases, delete all the code because they do not see the point in it due to the fact that there are no lines and WinAPI doesn't remain
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the attached prog1.zip and prog1.vmp.exe.zip samples, focusing on the virtualized or mutated OEP, empty DATA/RDATA sections, encrypted API calls, and the LLVM deobfuscation result. Reproduce the static-analysis behavior and review how RetDec handles these inputs; the issue does not define a concrete acceptance criterion or name a source file or test for the fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, reverse-engineering
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100