Error: failed to limit maximal memory to half of system RAM
- Dominant language
- C++
- Stars
- 8.6k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
after unzipped retdec-v3.2-windows-64b.zip from https://github.com/avast-tl/retdec/releases I run
```
O:\greensoft\retdec\bin>python3 retdec-archive-decompiler.py watc.dll
```
then I get
```
##### Checking if file is a Mach-O Universal static library...
##### Checking if file is an archive...
RUN: O:\greensoft\retdec\bin\retdec-ar-extractor O:\greensoft\retdec\bin\watc.dl
l --arch-magic
Not an archive, going to the next step.
##### Gathering file information...
RUN: O:\greensoft\retdec\bin\retdec-fileinfo -c O:\greensoft\retdec\bin\watc.dll
.c.json --similarity O:\greensoft\retdec\bin\watc.dll --no-hashes=all --crypto O
:\greensoft\retdec\bin\..\share\retdec\support\generic\yara_patterns\signsrch\si
gnsrch.yara --max-memory-half-ram
Input file : O:\greensoft\retdec\bin\watc.dll
File format : PE
File class : 32-bit
File type : DLL
Architecture : x86
Endianness : Little endian
Image base address : 0x10000000
Entry point address : 0x10002b14
Entry point offset : 0x2b14
Entry point section name : .text
Entry point section index: 0
Bytes on entry point : 558bec538b5d08568b750c578b7d1085f67509833d7c9d001000e
b2683fe01740583fe027522a1283f011085c07409575653
Detected tool : Microsoft Linker (6.0) (linker), combined heuristic
Detected tool : MSVC (6.0 debug) Visual Studio 6.0 (compiler), combin
ed heuristic
Original language : C++
Rich header offset : 0x80
Rich header key : 0x3f4d421a
Rich header signature : 000c1c7b00000001000b1fe800000001000e1c830000001500131
f6200000009000100000000005c000a1fe8
0000006200041fe800000001
##### Trying to unpack O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec
\bin\watc.dll-unpacked.tmp by using generic unpacker...
RUN: O:\greensoft\retdec\bin\retdec-unpacker O:\greensoft\retdec\bin\watc.dll -o
O:\greensoft\retdec\bin\watc.dll-unpacked.tmp --max-memory-half-ram
Failed to limit memory to half of system RAM!
##### Unpacking by using generic unpacker: failed
##### Trying to unpack O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec
\bin\watc.dll-unpacked.tmp by using UPX...
RUN: upx -d O:\greensoft\retdec\bin\watc.dll -o O:\greensoft\retdec\bin\watc.dll
-unpacked.tmp
upx: O:\greensoft\retdec\bin\watc.dll: NotPackedException: not packed by UPX
##### Unpacking by using UPX: nothing to do
##### Decompiling O:\greensoft\retdec\bin\watc.dll into O:\greensoft\retdec\bin\
watc.dll.c.backend.bc...
RUN: O:\greensoft\retdec\bin\retdec-bin2llvmir -provider-init -decoder -verify -
main-detection -idioms-libgcc -inst-opt -register -cond-branch-opt -syscalls -st
ack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -
remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -va
lue-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -do
mtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg
-instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-t
hreading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassoc
iate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombin
e -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-lo
ad-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -i
nstcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -me
mdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes
-globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine
-tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-
expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifyc
fg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-pro
pagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -lo
op-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loo
p-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars
-loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-i
nfo -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -ad
ce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge
-constprop -instnamer -domtree -instcombine -simple-types -stack-ptr-op-remove -
inst-opt -idioms -global-to-local -dead-global-assign -instcombine -phi2seq -val
ue-protect -disable-inlining -disable-simplify-libcalls -config-path O:\greensof
t\retdec\bin\watc.dll.c.json -max-memory-half-ram -o O:\greensoft\retdec\bin\wat
c.dll.c.backend.bc
Running phase: Initialization ( 0.01s )
Error: failed to limit maximal memory to half of system RAM
Error: Decompilation to LLVM IR failed
O:\greensoft\retdec\bin>
```
where the watc.dll is from https://drive.google.com/open?id=1EpPs7oAUleS2xjCSqDX0rfuHkWZ0A3yU
I have tested with some other windows application and I get the same error
Then if I delete `-indvars-loop-idiom`, ad replace `-constmerge-constprop` with `-constmerge`, the following can give me `watc.dll.c.backend.bc`, `watc.dll.c.backend.ll` and `watc.dll.c.frontend.dsm`
```
O:\greensoft\retdec\bin\retdec-bin2llvmir -provider-init -decoder -verify -main-detection -idioms-libgcc -inst-opt -register -cond-branch-opt -syscalls -stack -constants -param-return -local-vars -inst-opt -simple-types -generate-dsm -remove-asm-instrs -class-hierarchy -select-fncs -unreachable-funcs -inst-opt -value-protect -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -indvars -loop-idiom -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -constprop -instnamer -domtree -instcombine -instcombine -tbaa -targetlibinfo -basicaa -domtree -simplifycfg -domtree -early-cse -lower-expect -targetlibinfo -tbaa -basicaa -globalopt -mem2reg -instcombine -simplifycfg -basiccg -domtree -early-cse -lazy-value-info -jump-threading -correlated-propagation -simplifycfg -instcombine -simplifycfg -reassociate -domtree -loops -loop-simplify -lcssa -loop-rotate -licm -lcssa -instcombine -scalar-evolution -loop-simplifycfg -loop-simplify -aa -loop-accesses -loop-load-elim -lcssa -loop-deletion -memdep -gvn -memdep -sccp -instcombine -lazy-value-info -jump-threading -correlated-propagation -domtree -memdep -dse -dce -bdce -adce -die -simplifycfg -instcombine -strip-dead-prototypes -globaldce -constmerge -instnamer -domtree -instcombine -simple-types -stack-ptr-op-remove -inst-opt -idioms -global-to-local -dead-global-assign -instcombine -phi2seq -value-protect -disable-inlining -disable-simplify-libcalls -config-path O:\greensoft\retdec\bin\watc.dll.c.json -o O:\greensoft\retdec\bin\watc.dll.c.backend.bc
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the Windows command through retdec-archive-decompiler.py, then compare how retdec-unpacker and retdec-bin2llvmir handle --max-memory-half-ram. Trace the failure reported during unpacking and LLVM IR initialization. Done means the reproduced DLL decompiles without the memory-limit errors, or the limitation is clearly reported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- compilers, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100