avast / avast/retdec

Error: failed to limit maximal memory to half of system RAM

Open
#427 8 comments 0 reactions 0 assignees View on GitHub
bug O-windows P-run
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

after unzipped retdec-v3.2-windows-64b.zip from https://github.com/avast-tl/retdec/releases I run
```
O:\greensoft\retdec\bin>python3 retdec-archive-decompiler.py watc.dll
```
then I get
```
##### Checking if file is a Mach-O Universal static library...

##### Checking if file is an archive...
RUN: O:\greensoft\retdec\bin\retdec-ar-extractor O:\greensoft\retdec\bin\watc.dl
l --arch-magic
Not an archive, going to the next step.

##### Gathering file information...
RUN: O:\greensoft\retdec\bin\retdec-fileinfo -c O:\greensoft\retdec\bin\watc.dll
.c.json --similarity O:\greensoft\retdec\bin\watc.dll --no-hashes=all --crypto O
:\greensoft\retdec\bin\..\share\retdec\support\generic\yara_patterns\signsrch\si
gnsrch.yara --max-memory-half-ram
Input file : O:\greensoft\retdec\bin\watc.dll
File format : PE
File class : 32-bit
File type : DLL
Architecture : x86
Endianness : Little endian
Image base address : 0x10000000
Entry point address : 0x10002b14
Entry point offset : 0x2b14
Entry point section name : .text
Entry point section index: 0
Bytes on entry point : 558bec538b5d08568b750c578b7d1085f67509833d7c9d001000e
b2683fe01740583fe027522a1283f011085c07409575653
Detected tool : Microsoft Linker (6.0) (linker), combined heuristic
Detected tool : MSVC (6.0 debug) Visual Studio 6.0 (compiler), combin
ed heuristic
Original language : C++
Rich header offset : 0x80
Rich header key : 0x3f4d421a
Rich header signature : 000c1c7b00000001000b1fe800000001000e1c830000001500131
f6200000009000100000000005c000a1fe8
0000006200041fe800000001

##### Trying to unpack O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec
\bin\watc.dll-unpacked.tmp by using generic unpacker...
RUN: O:\greensoft\retdec\bin\retdec-unpacker O:\greensoft\retdec\bin\watc.dll -o
O:\greensoft\retdec\bin\watc.dll-unpacked.tmp --max-memory-half-ram
Failed to limit memory to half of system RAM!
##### Unpacking by using generic unpacker: failed

##### Trying to unpack O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec
\bin\watc.dll-unpacked.tmp by using UPX...
RUN: upx -d O:\greensoft\retdec\bin\watc.dll -o O:\greensoft\retdec\bin\watc.dll
-unpacked.tmp
upx: O:\greensoft\retdec\bin\watc.dll: NotPackedException: not packed by UPX
##### Unpacking by using UPX: nothing to do

##### Decompiling O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec\bin\
watc.dll.c.backend.bc...
RUN: O:\greensoft\retdec\bin\retdec-bin2llvmir -provider-init -decoder -verify -
main-detection -idioms-libgcc -inst-opt -register -cond-branch-opt -syscalls -st
ack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -
remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -va
lue-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -do
mtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg
-instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-t
hreading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassoc
iate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombin
e -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-lo
ad-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -i
nstcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -me
mdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes
-globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine
-tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-
expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifyc
fg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-pro
pagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -lo
op-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loo
p-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars
-loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-i
nfo -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -ad
ce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge
-constprop -instnamer -domtree -instcombine -simple-types -stack-ptr-op-remove -
inst-opt -idioms -global-to-local -dead-global-assign -instcombine -phi2seq -val
ue-protect -disable-inlining -disable-simplify-libcalls -config-path O:\greensof
t\retdec\bin\watc.dll.c.json -max-memory-half-ram -o O:\greensoft\retdec\bin\wat
c.dll.c.backend.bc
Running phase: Initialization ( 0.01s )
Error: failed to limit maximal memory to half of system RAM
Error: Decompilation to LLVM IR failed

O:\greensoft\retdec\bin>
```
where the watc.dll is from https://drive.google.com/open?id=1EpPs7oAUleS2xjCSqDX0rfuHkWZ0A3yU

I have tested with some other windows application and I get the same error

Then if I delete `-indvars-loop-idiom`, ad replace `-constmerge-constprop` with `-constmerge`, the following can give me `watc.dll.c.backend.bc`, `watc.dll.c.backend.ll` and `watc.dll.c.frontend.dsm`
```
O:\greensoft\retdec\bin\retdec-bin2llvmir -provider-init -decoder -verify -main-detection -idioms-libgcc -inst-opt -register -cond-branch-opt -syscalls -stack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -value-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -instnamer -domtree -instcombine -simple-types -stack-ptr-op-remove -inst-opt -idioms -global-to-local -dead-global-assign -instcombine -phi2seq -value-protect -disable-inlining -disable-simplify-libcalls -config-path O:\greensoft\retdec\bin\watc.dll.c.json -o O:\greensoft\retdec\bin\watc.dll.c.backend.bc
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Windows command through retdec-archive-decompiler.py, then compare how retdec-unpacker and retdec-bin2llvmir handle --max-memory-half-ram. Trace the failure reported during unpacking and LLVM IR initialization. Done means the reproduced DLL decompiles without the memory-limit errors, or the limitation is clearly reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
compilers, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.