avast / avast/retdec

retdec-llvmir2hll fails with "terminate called after throwing an instance of 'std::bad_alloc'"

Open
#232 12 comments 0 reactions 0 assignees View on GitHub
bug C-llvmir2hll P-run T-memory
Dominant language
C++
Stars
8.6k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

I'm trying to decompile a file with **3.2MB** size.

I closed all my applications and use: `ulimit -Sv 9863168` to limit memory (i also try with 4GB), but everytime i get this error below, to run properly i need to get more ram or is there any way i optimize this to get the final C source?

```bash
Running phase: removing functions prefixed with [__decompiler_undefined_function_] ( 168.85s )
Running phase: removing functions from standard libraries ( 194.89s )
Running phase: removing code that is not reachable in a CFG ( 194.96s )
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v5_405ddf = *(IntToPtrCastExpr((v0_405e01 + 1)))` -> skipping this edge
Running phase: signed/unsigned types fixing ( 199.13s )
Running phase: converting LLVM intrinsic functions to standard functions ( 237.54s )
Running phase: obtaining debug information ( 240.46s )
Running phase: alias analysis [simple] ( 240.61s )
Running phase: optimizations [normal] ( 242.87s )
-> running RemoveUselessCastsOptimizer ( 242.87s )
-> running UnusedGlobalVarOptimizer ( 244.68s )
-> running DeadLocalAssignOptimizer ( 247.27s )
Warning: out of memory; trying to recover
-> running SimpleCopyPropagationOptimizer ( 281.59s )
Warning: [NonRecursiveCFGBuilder] there is no node for an edge to `v5_405ddf = *(IntToPtrCastExpr((v0_405e01 + 1)))` -> skipping this edge
Warning: out of memory; trying to recover
-> running CopyPropagationOptimizer ( 366.28s )
Warning: out of memory; trying to recover
-> running AuxiliaryVariablesOptimizer ( 375.55s )
Warning: out of memory; trying to recover
-> running SimplifyArithmExprOptimizer ( 378.81s )
Warning: out of memory; trying to recover
-> running IfStructureOptimizer ( 379.43s )
Warning: out of memory; trying to recover
-> running LoopLastContinueOptimizer ( 379.63s )
Warning: out of memory; trying to recover
-> running PreWhileTrueLoopConvOptimizer ( 379.75s )
Warning: out of memory; trying to recover
-> running WhileTrueToForLoopOptimizer ( 381.69s )
Warning: out of memory; trying to recover
-> running WhileTrueToWhileCondOptimizer ( 381.69s )
Warning: out of memory; trying to recover
-> running IfBeforeLoopOptimizer ( 381.70s )
Warning: out of memory; trying to recover
-> running LLVMIntrinsicsOptimizer ( 381.70s )
Warning: out of memory; trying to recover
-> running VoidReturnOptimizer ( 381.72s )
Warning: out of memory; trying to recover
-> running BreakContinueReturnOptimizer ( 381.73s )
Warning: out of memory; trying to recover
-> running BitShiftOptimizer ( 381.73s )
Warning: out of memory; trying to recover
-> running DerefAddressOptimizer ( 381.74s )
Warning: out of memory; trying to recover
-> running EmptyArrayToStringOptimizer ( 381.74s )
-> running BitOpToLogOpOptimizer ( 381.74s )
Warning: out of memory; trying to recover
-> running SimplifyArithmExprOptimizer ( 381.75s )
Warning: out of memory; trying to recover
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
0 retdec-llvmir2hll 0x000000000095a6ce
1 retdec-llvmir2hll 0x000000000095a81c
2 libpthread.so.0 0x00007fda3feaf390
3 libc.so.6 0x00007fda3ee3b428 gsignal + 56
4 libc.so.6 0x00007fda3ee3d02a abort + 362
5 libstdc++.so.6 0x00007fda3f77e84d _ZN9__gnu_cxx27__verbose_terminate_handlerEv + 365
6 libstdc++.so.6 0x00007fda3f77c6b6
7 libstdc++.so.6 0x00007fda3f77c701
8 libstdc++.so.6 0x00007fda3f77c919
9 libstdc++.so.6 0x00007fda3f77cebc
10 retdec-llvmir2hll 0x00000000005cc26f
11 retdec-llvmir2hll 0x00000000006f7ff6
12 retdec-llvmir2hll 0x0000000000680329
13 retdec-llvmir2hll 0x000000000067dfef
14 retdec-llvmir2hll 0x0000000000486b83
15 retdec-llvmir2hll 0x00000000004887ba
16 retdec-llvmir2hll 0x0000000000b46514
17 retdec-llvmir2hll 0x0000000000487c53
18 retdec-llvmir2hll 0x0000000000410691
19 libc.so.6 0x00007fda3ee26830 __libc_start_main + 240
20 retdec-llvmir2hll 0x00000000004805b9
Stack dump:
0. Program arguments: /home/laerte/reverse/retdec-tool/bin/retdec-llvmir2hll -target-hll=c -var-renamer=readable -var-name-gen=fruit -var-name-gen-prefix= -call-info-obtainer=optim -arithm-expr-evaluator=c -validate-module -llvmir2bir-converter=orig -o /home/laerte/reverse/retdec-tool/bin/thor.c /home/laerte/reverse/retdec-tool/bin/thor.c.backend.bc -enable-debug -emit-debug-comments -config-path=/home/laerte/reverse/retdec-tool/bin/thor.c.json
1. Running pass 'Decompiler' on module '/home/laerte/reverse/retdec-tool/bin/thor.c.backend.bc'.
./retdec-decompiler.sh: linha 922: 6253 Abortado "$LLVMIR2HLL" "${LLVMIR2HLL_PARAMS[@]}"
Error: Decompilation of file '/home/laerte/reverse/retdec-tool/bin/thor.c.backend.bc' failed
```
### Setup:
Ubuntu 16.04.3 LTS
Intel(R) Core(TM) i5-4460 CPU @ 3.20GHz (4th Gen)
8GB ram

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the retdec-llvmir2hll command on the reported .backend.bc input and inspect the optimizer phase where allocation fails; the wrapper reports the failure at retdec-decompiler.sh line 922. The report names no source file or test, so done means isolating a reproducible cause and defining a verified fix or documented resource limitation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, reverse-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.