avast / avast/retdec

The problem with static analysis of VMP-like protectors. CODE && DATA detect. ROP problems

Open
#1,155 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

The problem lies in the obvious disadvantage of static analysis alone. I'll tell you everything now:
We take an example(MsgB) from VMP 2.13.2
Clear example: [prog1.zip](https://github.com/avast/retdec/files/11346990/prog1.zip)
VMP 2.13.2 example: [prog1.vmp.exe.zip](https://github.com/avast/retdec/files/11346992/prog1.vmp.exe.zip)

If you go into the example with VMP, you will see that the sections with variables are empty, and API calls are encrypted

VMP 2.13.2 example:

VMP 2.13.2 RDATA

![image](https://user-images.githubusercontent.com/120374383/234963505-9db83a74-2a1e-4acf-af66-033950f8fe48.png)

VMP 2.13.2 DATA

![image](https://user-images.githubusercontent.com/120374383/234963631-265b53c7-0a29-4b07-a389-84845be20d1f.png)

VMP 2.13.2 RDATA API

![image](https://user-images.githubusercontent.com/120374383/234963797-79a18ee7-9b08-4a76-bf59-5f0fc6ab060b.png)

Clear example:

Clear RDATA

![image](https://user-images.githubusercontent.com/120374383/234964119-02f4ac7c-3e89-4d79-89c2-9d6a39fd362c.png)

Clear DATA

![image](https://user-images.githubusercontent.com/120374383/234964175-4edc166a-744d-4039-b063-15e3a2aa92b1.png)

Clear RDATA API

![image](https://user-images.githubusercontent.com/120374383/234964300-5cb92080-03f1-43ff-8179-a1033b99c9c3.png)

;---END

It's contains both example in RDATA API
On the first line - MessageBoxA

On the second line - ExitProcess

Also, do not pay attention to the inscription in MsgB, first I packed VMP 3.8.1, but I thought it was better to start with something old

As you can see in the case of VMP 2.13.2, the VMP loader deletes everything, and it is impossible to perform static analysis of the virtualized or mutated EntryPoint function, since our OEP is essentially virtualized/mutated. We can't even dump and start the analysis in static, since the OEP is virtualized/mutated. I'm still thinking about solving this problem. If anyone has any ideas, I will be glad if you can share it

p.s.
When trying to deobfuscate the example with VMP, LLVM deletes the entire code (it does the right thing). LLVM, or rather its optimization phases, delete all the code because they do not see the point in it due to the fact that there are no lines and WinAPI doesn't remain

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the attached prog1.zip and prog1.vmp.exe.zip samples, focusing on the virtualized or mutated OEP, empty DATA/RDATA sections, encrypted API calls, and the LLVM deobfuscation result. Reproduce the static-analysis behavior and review how RetDec handles these inputs; the issue does not define a concrete acceptance criterion or name a source file or test for the fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, reverse-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.